return-to-ritherdon
/
ritherdon-charts
1
0
Fork 0
Code Releases 1 Activity
Browse Source

update README.org file.

master
Craig Oates 1 year ago
parent
cc66f7359f
commit
df2bac4ab4
1 changed files with 69 additions and 44 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 113
      README.org

113
README.org
Unescape View File

@ -18,13 +18,14 @@ Gaps in the Line]] exhibition by [[http://www.nicolaellis.com][Nicola Ellis]], h
Manchester, U.K. Manchester, U.K.
This project ties into a larger collection of software projects related to the This project ties into a larger collection of software projects related to the
/Personal Flash in Real-Time/ artworks. Which, in turn, is a small piece of the /Personal Flash in Real-Time/ artworks. And, those artworks are a small piece of
much larger [[https://www.nicolaellisandritherdon.com/][Return to Ritherdon]] project (devised and completed by Nicola the much larger [[https://www.nicolaellisandritherdon.com/][Return to Ritherdon]] project (devised and completed by Nicola
Ellis). For more information on the software-side of the /Personal Flash in Ellis). For more information on the artworks and where they sit in the larger
Real-Time/ artworks, please use the links below: project, please use the links below:
- [[https://git.abbether.net/return-to-ritherdon/rtr-docs][rtr-docs]] (The documentation repository for all the /Person Flash in Real-Time/ - [[https://git.abbether.net/return-to-ritherdon/rtr-docs][rtr-docs]] (The documentation repository for all the /Person Flash in Real-Time/
software projects) software projects, with an overview of how they tie into the Return to
Ritherdon project)
- [[https://git.abbether.net/return-to-ritherdon][Return to Ritherdon Org. Page]] (The 'home page' for the /Return to Ritherdon/ - [[https://git.abbether.net/return-to-ritherdon][Return to Ritherdon Org. Page]] (The 'home page' for the /Return to Ritherdon/
project on this site, containing a list of all the publicly available project on this site, containing a list of all the publicly available
repositories) repositories)
@ -35,16 +36,16 @@ manufacturing electrical enclosures and other related products. So, if you have
spent any time in the U.K. and seen one of those green electrical boxes lurking spent any time in the U.K. and seen one of those green electrical boxes lurking
on a street corner, there is a good chance these folks made it. on a street corner, there is a good chance these folks made it.
*NOTE: This project does not contain documentation in the [[https://git.abbether.net/return-to-ritherdon/rtr-docs][rtr-docs]] repository.* It *NOTE: This project does not contain documentation in the [[https://git.abbether.net/return-to-ritherdon/rtr-docs][rtr-docs]] repository.*
is a self-contained mini-project which is not directly related to the /Personal This is a self-contained mini-project which is not /directly related/ to the
Flash in Real-Time/ artworks. /Personal Flash in Real-Time/ artworks.
*** Examples/Screenshots *** Examples/Screenshots
At the time of writing, the scripts in this repository produce over one-hundred At the time of writing, the scripts in this repository produce over one-hundred
charts/files. So, here are a selection of the types of charts produced after charts/files. So, here are a selection of the types of charts produced after
processing the data in =data/lm1-exhibiton-all.csv= and processing the data in the =data/lm1-exhibiton-all.csv= and
=/data/lm2-exhibition-all.csv=. =/data/lm2-exhibition-all.csv= files.
[[file:assets/daily-totals.png]] [[file:assets/daily-totals.png]]
@ -59,14 +60,14 @@ processing the data in =data/lm1-exhibiton-all.csv= and
** Overview of the /Personal Flash in Real-Time/ Artworks ** Overview of the /Personal Flash in Real-Time/ Artworks
/Personal Flash in Real-Time/ consists of two artworks, named /Personal Flash in /Personal Flash in Real-Time/ consists of two artworks, named /Personal Flash in
Real-Time (Andy)/ and /Personal Flash in Real-Time (Tony)/. Each one measured the Real-Time (Andy)/ and /Personal Flash in Real-Time (Tony)/. Each one measured
light in the welding booths in the Ritherdon Factory and forwarded those the light in the welding booths in the Ritherdon Factory and forwarded those
readings on to a server running in Amazon's 'cloud' -- see [[https://aws.amazon.com/][Amazon Web Services]] readings on to a server running in Amazon's 'cloud' -- see [[https://aws.amazon.com/][Amazon Web Services]]
(/AWS/) for more information. While this was happening, two sets of lights (AWS) for more information. While this was happening, two sets of lights,
residing in Castlefield Gallery would turn on and off whenever the system residing in Castlefield Gallery, would turn on and off whenever the system
detected someone welding in Ritherdon. The would happen because the Relays, detected someone welding in Ritherdon. The would happen because the Relays,
controlling the lights would receive the latest Light Meter readings taken in controlling the lights would receive the latest Light Meter readings taken in
Ritherdon via the server (/AWS/). Ritherdon via the server (AWS).
The (AWS) server stored every reading taken in a SQLite database and this The (AWS) server stored every reading taken in a SQLite database and this
project pokes and prods at the data -- to plot charts/graphs. project pokes and prods at the data -- to plot charts/graphs.
@ -79,10 +80,10 @@ project pokes and prods at the data -- to plot charts/graphs.
2. The =separator.sh= and =totalilator.sh= scripts split the .csv files, mentioned 2. The =separator.sh= and =totalilator.sh= scripts split the .csv files, mentioned
above, into smaller files in an attempt to make them (.csv files) easier to above, into smaller files in an attempt to make them (.csv files) easier to
work with on average hardware. work with on average hardware.
1. On top of that,I have only committed the .csv files mentioned in point 1 1. On top of that,I have only committed the CSV files mentioned in point 1
to the repository as a means to reduce the clutter in the repositories Git to the repository as a means to reduce the clutter in the repositories Git
commit history. commit history.
2. You will need to split the .csv files up yourself after you have cloned 2. You will need to split the CSV files up yourself after you have cloned
the repository, using the scripts mentioned in point 2. the repository, using the scripts mentioned in point 2.
3. *The database containing the actual data is not included with this repository.* 3. *The database containing the actual data is not included with this repository.*
1. The database used for the /No Gaps in the Line/ exhibition is approximately 1. The database used for the /No Gaps in the Line/ exhibition is approximately
@ -92,28 +93,28 @@ project pokes and prods at the data -- to plot charts/graphs.
(13^th June, 2021) and 2021-08-01 (1^st August, 2021) for both Light Meters (13^th June, 2021) and 2021-08-01 (1^st August, 2021) for both Light Meters
(the length of the exhibition): There is (not much) more (test) data in the (the length of the exhibition): There is (not much) more (test) data in the
database but the data selected/exported seemed the most appropriate decision. database but the data selected/exported seemed the most appropriate decision.
5. I chose to work with .csv files out of convenience more than anything else -- 5. I chose to work with CSV files out of convenience more than anything else --
it is the easiest format to export the data to from the SQlite database. it is the easiest format to export the data to from the SQlite database.
6. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (/rg/), also, out of convenience, they were 6. I used [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)][Bash]], [[https://en.wikipedia.org/wiki/AWK][Awk]] and [[https://github.com/BurntSushi/ripgrep][Ripgrep]] (rg), also, out of convenience, they were
already on my computer. already on my computer.
7. I used [[https://bokeh.org/][Bokeh]] because I have already used it and it is the only thing I know 7. I used [[https://bokeh.org/][Bokeh]] because I have already used it and it is the only thing I know
which can create interactive charts as individual HTML files, which I can which can create interactive charts as individual HTML files, which I can
share with anyone not comfortable with computers. share with anyone not comfortable with computers.
8. I used [[https://www.python.org/][Python]] because of Bokeh. 8. I used [[https://www.python.org/][Python]] because of Bokeh.
9. Overall, Nicola wants to work with the charts this data produces so 9. Overall, Nicola wants to work with the charts produced by this data so any
any decisions made should be in service to that end. decisions made should be in service to that end.
10. I have taken a hard-coded approach to filenames with the code because the 10. I have taken a hard-coded approach to filenames with the code because the
code is not the main objective here, the charts are; *Long-term flexibility code is not the main objective here, the charts are; In other words,
and maintenance is not a concern here.* *long-term flexibility and maintenance is not a concern here.*
** Set-Up and Using the Code ** Set-Up and Using the Code
Open your terminal, making sure you are in the directory you want the repository Open your terminal, making sure you are in the directory you want the repository
cloned to. cloned to.
*The Bash (/.sh/) scripts need calling before the Python (/.py/) ones.* You need to *The Bash (.sh) scripts need calling before the Python (.py) ones.* You need to
process the =lm1-exhibition-all.csv= and =lm2-exhibition-all.csv= file first because process the =lm1-exhibition-all.csv= and =lm2-exhibition-all.csv= files first because
the Python (/.py/) scripts assumes certain files are already in the =/data= the Python (.py) scripts assumes certain files are already in the =/data=
directory. directory.
#+begin_src shell #+begin_src shell
@ -143,8 +144,8 @@ directory.
#+end_src #+end_src
When the packages have finished installed (via ~pip~), you should be ready to When the packages have finished installed (via ~pip~), you should be ready to
go. From there, you can simply call the Python (/.py/) scripts (from terminal with go. From there, you can simply call the Python (.py) scripts (from terminal with
the venv. activated). For example, the =venv= activated). For example,
#+begin_src shell #+begin_src shell
# Make sure you are in the terminal with the virtual-environment activated... # Make sure you are in the terminal with the virtual-environment activated...
@ -155,22 +156,30 @@ the venv. activated). For example,
# Output from script... # Output from script...
#+end_src #+end_src
When you have finished, you will need to deactivate your Python When you have finished, you will need to deactivate the virtual-environment. You
virtual-environment. You can do that by entering ~disable~ in your terminal. You can do that by entering ~disable~ in your terminal. You should see the ~(venv)~
should see the ~(venv)~ part of your prompt removed. part of your prompt removed.
#+begin_src shell #+begin_src shell
# Before you disable your Python virtual-environment (venv)... # Before you disable your Python's virtual-environment (venv)...
# (venv) yourname@yourpc:~/local-dev/ritherdon-charts$ # (venv) yourname@yourpc:~/local-dev/ritherdon-charts$
# Disable your Python venv. # Disable your Python venv.
disable disable
# After you have disabled your Python virtual-environment (venv)... # After you have disabled your Python's virtual-environment (venv)...
# yourname@yourpc:~/local-dev/ritherdon-charts$ # yourname@yourpc:~/local-dev/ritherdon-charts$
#+end_src #+end_src
** Working with the Files/Data Produced After Running the Project's Scripts From here, you can either write you own scripts to form new charts or just play
with the CSV files in something like Microsoft Excel or Libre Office Calc.
** Working with the Files/Data Produced After Running the Project's Bash Scripts
#+begin_quote
=/data= stores the CSV files and =/output= stores the charts. Run ~./separator.sh~
to get started.
#+end_quote
When you clone the repository, you will find the =/data= directory will have the When you clone the repository, you will find the =/data= directory will have the
following layout, following layout,
@ -188,9 +197,13 @@ data
0 directories, 2 files 0 directories, 2 files
#+end_src #+end_src
*The =/output= directory not exist before you run the scripts.* This =/data= directory is responsible for storing the /raw/ data (I.E. the CSV
files). The charts, created via the Python (.py) scripts, reside in the
=/output= directory.
*The =/output= directory should not exist until you run ~./separator.sh~.*
After you run the Bash scripts (~seperator.sh~ and ~totalilator.sh~), you should see After you run the Bash scripts (~./seperator.sh~ and ~./totalilator.sh~), you should see
something like the following in the =/data= directory, something like the following in the =/data= directory,
#+begin_src shell :results code #+begin_src shell :results code
@ -201,12 +214,12 @@ something like the following in the =/data= directory,
#+begin_src shell #+begin_src shell
data data
├── light-meter-1 ├── light-meter-1
│   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021. │   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021.
│   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021. │   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021.
│   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021. │   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021.
│   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021. │   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021.
# More files and folder here... # More files and folder here...
│   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021 │   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021
│   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021. │   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021.
│ │
├── light-meter-1-daily-totals.csv # Total number of readings recorded for each day. ├── light-meter-1-daily-totals.csv # Total number of readings recorded for each day.
@ -215,12 +228,12 @@ something like the following in the =/data= directory,
│   └── 2021-07 # Directory containing files with hourly totals (per day) for July. │   └── 2021-07 # Directory containing files with hourly totals (per day) for July.
│ │
├── light-meter-2 ├── light-meter-2
│   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021. │   ├── 2021-06-13 # Directory of readings taken per hour taken on 13/06/2021.
│   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021. │   ├── 2021-06-13.csv # File containing all the reading taken on 13th June 2021.
│   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021. │   ├── 2021-06-14 # Directory of readings taken per hour taken on 14/06/2021.
│   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021. │   ├── 2021-06-14.csv # File containing all the reading taken on 14th June 2021.
# More files and folders here... # More files and folders here...
│   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021. │   ├── 2021-07-30 # Directory of readings taken per hour taken on 30/07/2021.
│   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021. │   └── 2021-07-30.csv # File containing all the reading taken on 30th July 2021.
│ │
├── light-meter-2-daily-totals.csv # Total number of readings recorded for each day. ├── light-meter-2-daily-totals.csv # Total number of readings recorded for each day.
@ -235,7 +248,7 @@ something like the following in the =/data= directory,
#+end_src #+end_src
*The =/output= directory should now exist in the project's root directory -- with *The =/output= directory should now exist in the project's root directory -- with
nothing in it.* This is where the charts will go, after running the Python (/.py/) nothing in it.* This is where the charts will go, after running the Python (.py)
scripts. The ~separator.sh~ script created it. The project's root directory should scripts. The ~separator.sh~ script created it. The project's root directory should
now look like the following, now look like the following,
@ -256,7 +269,7 @@ now look like the following,
├── LICENSE ├── LICENSE
├── lm1-hourly-totals.py ├── lm1-hourly-totals.py
├── lm2-hourly-totals.py ├── lm2-hourly-totals.py
├── output # This should now exist and charts go in here... ├── output # *NEW* This should now exist and charts go in here...
├── README.org ├── README.org
├── requirements.txt ├── requirements.txt
├── separator.sh ├── separator.sh
@ -266,3 +279,15 @@ now look like the following,
4 directories, 13 files 4 directories, 13 files
#+end_src #+end_src
From here, you can either create some charts with the Python (.py) scripts and
take a look at them in =/output= or open the CSV files in =/data= and inspect the
data.
*** Overview of the CSV files in =/data=
I need to write this...
*** Overview of Scripts and Charts
I need to write this...

Write Preview
Loading…
Cancel
Save