|
|
|
# Return to Ritherdon: Midpoint
|
|
|
|
|
|
|
|
Midpoint is one of three smaller projects which make-up 'Personal
|
|
|
|
Flash in Real-Time' -- which is one artwork within the main 'Return to
|
|
|
|
Ritherdon' project. You should view the three smaller projects as one
|
|
|
|
project. And, for the purpose of this documentation, I will refer to
|
|
|
|
'Personal Flash in Real-Time' as a system instead of an artwork.
|
|
|
|
|
|
|
|
For more information on the 'Return to Ritherdon' project, use the
|
|
|
|
following link:
|
|
|
|
|
|
|
|
- [Return to Ritherdon Overview](https://git.abbether.net/return-to-ritherdon/rtr-docs)
|
|
|
|
|
|
|
|
## 'Personal Flash in Real-Time': Project Overview
|
|
|
|
|
|
|
|
The overall system ('Personal Flash in Real-Time') consists of three
|
|
|
|
separate/smaller projects. You should not view them as individual
|
|
|
|
pieces within the overall project. From an artwork point-of-view,
|
|
|
|
'Personal Flash in Real-Time' is one piece. The name of the software
|
|
|
|
projects are 'Light Meter' (which is this one), 'Midpoint' and
|
|
|
|
'Relay'. All three projects reside in their own git repositories. You
|
|
|
|
can find the repositories at the following links:
|
|
|
|
|
|
|
|
- [Light Meter](https://git.abbether.net/return-to-ritherdon/light-meter)
|
|
|
|
- [Mid-Point](https://git.abbether.net/return-to-ritherdon/midpoint)
|
|
|
|
- [Relay](https://git.abbether.net/return-to-ritherdon/relay)
|
|
|
|
|
|
|
|
The (technical) aim of the project is to turn a set of lights on at
|
|
|
|
the gallery when the welding machines are active in the welding booths
|
|
|
|
at Ritherdon. The solution we arrived at was a three-stage
|
|
|
|
process. The stages are as follows:
|
|
|
|
|
|
|
|
1. Monitor the light levels in the welding booths at Ritherdon and
|
|
|
|
send that information to a sever (Light Meter).
|
|
|
|
2. Receive the light readings and store them in a database and make
|
|
|
|
them available for others to access (Midpoint).
|
|
|
|
3. Have the lights installed at the gallery connected to wi-fi enabled
|
|
|
|
relays which request the latest light readings from the server. If
|
|
|
|
the readings are above a certain threshold, have the light in the
|
|
|
|
gallery turn on (otherwise, turn off). The relays are responsible
|
|
|
|
for turning the lights on and off (Relay).
|
|
|
|
|
|
|
|
Each step should require no human intervention.
|
|
|
|
|
|
|
|
For more information on how each project accomplishes its task, please
|
|
|
|
use the (repo.) links above. Otherwise, here is an diagram to help
|
|
|
|
explain the three stages mentioned above.
|
|
|
|
|
|
|
|
![Main Project Architecture](attachments/main-proj-architecture.png)
|
|
|
|
|
|
|
|
## Hardware Specifications
|
|
|
|
|
|
|
|
There are no specific hardware requirements. If you can get an machine
|
|
|
|
to run a server (E.G. Nginx or Apache), you should be good to
|
|
|
|
go. Unfortunately, the scope of the project meant the amount of
|
|
|
|
support for hardware and software is limited to an x86 Linux based
|
|
|
|
machine running Nginx. I decided to run a virtual machine on Amazon
|
|
|
|
Web Services.
|
|
|
|
|
|
|
|
- [Amazon Web Services](https://aws.amazon.com/) (A.W.S.)
|
|
|
|
|
|
|
|
## Software Requirements
|
|
|
|
|
|
|
|
I have set-up the server to run on Linux (I.E. Ubuntu) with an Nginx
|
|
|
|
server -- on an x86 machine. Any other variation on that
|
|
|
|
(I.E. Windows), you will be on your own. The reason why is because I
|
|
|
|
have not tested it. How you acquire the hardware is up to you. For the
|
|
|
|
intents of the project, I used a virtual machine on Amazon Web
|
|
|
|
Services. Links for various parts of the set-up are below:
|
|
|
|
|
|
|
|
- [https://ubuntu.com/](https://ubuntu.com/)
|
|
|
|
- [Nginx](https://www.nginx.com/)
|
|
|
|
- [Python Flask](https://flask.palletsprojects.com/en/2.0.x/)
|
|
|
|
- [Gunicorn](https://gunicorn.org/)
|
|
|
|
- [Supervisor](https://pypi.org/project/supervisor/) (Python)
|
|
|
|
- [SQLite Database](https://sqlite.org/index.html)
|
|
|
|
- [SQL Alchemy](https://www.sqlalchemy.org/) (O.R.M.)
|
|
|
|
- [Swagger REST API](https://swagger.io/)
|
|
|
|
|
|
|
|
## Software Set-up
|
|
|
|
|
|
|
|
Below is an overview of the files and directory structure.
|
|
|
|
|
|
|
|
```shell
|
|
|
|
app/
|
|
|
|
├── api.py <-------- REST API entry point.
|
|
|
|
├── app.py <------- 'Main' program entry point.
|
|
|
|
├── build_database.py <-- Builds database (use first).
|
|
|
|
├── config.py <----- Program config's for Flask and Swagger.
|
|
|
|
├── models <-------- Objects which database rows are mapped to.
|
|
|
|
├── __pycache__
|
|
|
|
├── readings.db <--- The SQLite database.
|
|
|
|
├── services <------ 'Business logic' for api.py entry points.
|
|
|
|
├── static <------- Images/Favicons/Style Sheets.
|
|
|
|
├── swagger.yml <--- YAML config. file for Swagger (REST API).
|
|
|
|
└── templates <----- HTML views.
|
|
|
|
```
|
|
|
|
|
|
|
|
### The Program (Python and Virtual Environment)
|
|
|
|
|
|
|
|
**I have omitted instructions on creating user accounts beyond the
|
|
|
|
defaults** because that comes with a fair bit of context specific
|
|
|
|
information which is beyond the scope of this documentation. On top of
|
|
|
|
that, the project's budget did not allow for extended research on
|
|
|
|
various Cloud Platforms and server architecture, in-general.
|
|
|
|
|
|
|
|
After you have established a V.M. on A.W.S. or your own server, you
|
|
|
|
will need to clone the repository (**I am assuming this is via SSH**).
|
|
|
|
|
|
|
|
```bash
|
|
|
|
# Adjust the directories to how you prefer it. These are just my defaults.
|
|
|
|
mkdir www
|
|
|
|
cd www
|
|
|
|
git clone http://git.abbether.net/return-to-ritherdon/midpoint.git
|
|
|
|
cd midpoint
|
|
|
|
```
|
|
|
|
|
|
|
|
From there, you will need to create a Python virtual environment,
|
|
|
|
|
|
|
|
```bash
|
|
|
|
# Adjust the path to suit to environment.
|
|
|
|
python3 -m venv ~/repos/midpoint/venv
|
|
|
|
```
|
|
|
|
|
|
|
|
As an aside, I created a virtual environment with `python3 -m venv
|
|
|
|
~/www/midpoint/my-env` whilst in development. This environment might
|
|
|
|
still be around the production server, used during the
|
|
|
|
exhibition. **If you are completing a fresh install, this is not
|
|
|
|
relevant to you. Please ignore.**
|
|
|
|
|
|
|
|
|
|
|
|
### The Server
|
|
|
|
|
|
|
|
The host name for the server used during the exhibition was
|
|
|
|
`piapi`. So, I will use that in the documentation below. **Please make
|
|
|
|
sure you replace `piapi` with the host name you are using.**
|
|
|
|
|
|
|
|
I used Nginx and Gunicorn for the exhibition so that is all this
|
|
|
|
documentation will focus on. Midpoint should work on other servers
|
|
|
|
(like Apache) but I have not tested it.
|
|
|
|
|
|
|
|
The Nginx configuration file is located at:
|
|
|
|
|
|
|
|
- `/etc/nginx/sites-enabled/piapi`
|
|
|
|
|
|
|
|
You can access the file with `sudo nano
|
|
|
|
/etc/nginx/sites-enabled/piapi` when logged into the server.
|
|
|
|
|
|
|
|
Because Nginx is responsible for the static side of the website,
|
|
|
|
Gunicorn is needed to operate the Python side of it. Nginx knows
|
|
|
|
how/when to pass control over to Gunicorn because it is configured in
|
|
|
|
the … sites-enables/piapi configuration file (mentioned above). With
|
|
|
|
that said, Gunicorn still needs to be set-up for everything to run
|
|
|
|
properly. It is worth pointing out, the "static" part of the site will
|
|
|
|
still run when Gunicorn is not running but that is a practically
|
|
|
|
useless feature. To run Gunicorn, enter the following command into the
|
|
|
|
terminal (of logged in V.M),
|
|
|
|
|
|
|
|
```bash
|
|
|
|
# You can adjust the '-w 3' if you have a more powerful server. Also,
|
|
|
|
the 'connext_app' related to the use of the Swagger API code.
|
|
|
|
gunicorn -w 3 server:connex_app
|
|
|
|
```
|
|
|
|
|
|
|
|
The `-w 3` bit refers to the total number of workers (processors)
|
|
|
|
Gunicorn will use whilst running. The general rule to work that out it
|
|
|
|
/(number of processors * 2) + 1. At the time of writing, the virtual
|
|
|
|
machine used for the exhibition is a very basic one-core machine so
|
|
|
|
that is why three workers is used.
|
|
|
|
|
|
|
|
One problem with Gunicorn is it blocks the terminal when using it in
|
|
|
|
its default way. To get around this, I have install Supervisor to
|
|
|
|
manage it. Supervisor handles the auto-restart after a crash and
|
|
|
|
background-task management of it too. To install it, run the following
|
|
|
|
command,
|
|
|
|
|
|
|
|
```bash
|
|
|
|
sudo apt install supervisor
|
|
|
|
```
|
|
|
|
|
|
|
|
When Supervisor is installed, you need to create a config. file. The
|
|
|
|
file for this project is stored at,
|
|
|
|
|
|
|
|
- `/etc/supervisor/conf.d/spiapi.conf`
|
|
|
|
|
|
|
|
The log files (specified in the .conf) file are,
|
|
|
|
|
|
|
|
- `/var/log/midpoint/midpoint.err.log`
|
|
|
|
- `/var/log/midpoint/midpoint.out.log`
|
|
|
|
|
|
|
|
You will need to recreate the directory they are stored in if you are
|
|
|
|
installing this site on a new system. The easiest way to do that is to
|
|
|
|
run the following command,
|
|
|
|
|
|
|
|
```bash
|
|
|
|
sudo mkdir -p /var/log/midpoint
|
|
|
|
sudo touch /var/log/midpoint/midpoint.err.log
|
|
|
|
sudo touch /var/log/midpoint/midpoint.out.log
|
|
|
|
```
|
|
|
|
|
|
|
|
When you have finished making these changes, you will need to restart
|
|
|
|
Supervisor. To do that, run the following command,
|
|
|
|
|
|
|
|
```bash
|
|
|
|
sudo supervisor reload
|
|
|
|
```
|
|
|
|
|
|
|
|
It might take a little while for the service to restart so if you
|
|
|
|
still see "Bad Gateway" messages in the browser, that might be why.
|
|
|
|
|
|
|
|
You might find Gunicorn is not installed in the virtual-environment
|
|
|
|
(if you have decided to not install one or you simply forget to
|
|
|
|
activate like I have a habit of doing), it that is the case, you
|
|
|
|
should find it at,
|
|
|
|
|
|
|
|
- `home/ubuntu.local/bin/gunicorn`
|
|
|
|
|
|
|
|
This might, also, mean you need to use `apt` and not `pip3`.
|
|
|
|
|
|
|
|
The config. file for Supervisor should look something like the
|
|
|
|
following, (remember to adjust the places where `ubuntu` is with your
|
|
|
|
servers account name -- if it differs from `ubuntu`)
|
|
|
|
|
|
|
|
```bash
|
|
|
|
[program:midpoint]
|
|
|
|
directory=/home/ubuntu/www/midpoint/app
|
|
|
|
command=/home/ubuntu/.local/bin/gunicorn -w 3 server:connex_app
|
|
|
|
user=ubuntu
|
|
|
|
autostart=true
|
|
|
|
autorestart=true
|
|
|
|
stopasgroup=true
|
|
|
|
killasgroup=true
|
|
|
|
stderr_logfile=/var/log/midpoint/midpoint.err.log
|
|
|
|
stdout_logfile=/var/log/midpoint/midpoint.out.log
|
|
|
|
```
|
|
|
|
|
|
|
|
The config. file for Nginx should look like the following,
|
|
|
|
|
|
|
|
```bash
|
|
|
|
server {
|
|
|
|
listen 80;
|
|
|
|
server_name 35.176.235.94;
|
|
|
|
|
|
|
|
location /static {
|
|
|
|
alias /home/ubuntu/www/midpoint/app/static;
|
|
|
|
}
|
|
|
|
|
|
|
|
location / { proxy_pass http://localhost:8000; include
|
|
|
|
/etc/nginx/proxy_params; proxy_redirect off; } }
|
|
|
|
```
|
|
|
|
|
|
|
|
**I did not set-up the server to use HTTPS because it was outside the
|
|
|
|
scope of the project.** Again, replace `ubuntu` with your servers
|
|
|
|
username.
|
|
|
|
|
|
|
|
### Flask & Swagger
|
|
|
|
|
|
|
|
To access the A.P.I. user-interface (via Swagger), enter
|
|
|
|
|
|
|
|
-`http://0.0.0.0:5000/api/ui/`
|
|
|
|
|
|
|
|
- [Real Python](https://realpython.com/flask-connexion-rest-api/#using-connexion-to-add-a-rest-api-endpoint)
|
|
|
|
|
|
|
|
Real Python provides a good walk-through for setting-up a website using
|
|
|
|
the Flask framework. Real Python, also, provides guides on creating
|
|
|
|
websites with Python and its various website frameworks.
|
|
|
|
|
|
|
|
I have imported the `connexion` project into this one. By doing this,
|
|
|
|
I can use Swagger and its U.I. When you have the server running, go to
|
|
|
|
`localhost:5000/api/swagger`. When there, you can test the A.P.I. out
|
|
|
|
by sending it "gets" and "posts".
|
|
|
|
|
|
|
|
![Swagger Screenshot](attachments/swagger-ui.png)
|
|
|
|
|
|
|
|
The Swagger endpoint are configured in a YAML file at,
|
|
|
|
|
|
|
|
- [midpoint/app/swagger.yml](https://git.abbether.net/return-to-ritherdon/midpoint/src/branch/unstable/app/swagger.yml)
|
|
|
|
|
|
|
|
### Database and O.R.M.
|
|
|
|
|
|
|
|
The project uses a SQLite database to store light reading and any
|
|
|
|
device status updates. To write to the database, I used SQL Alchemy as
|
|
|
|
a Object Relational Mapper (O.R.M). The database is called,
|
|
|
|
|
|
|
|
- `readings.db`
|
|
|
|
|
|
|
|
Before you can start the program, you will need to build the
|
|
|
|
database. To do that, you will need to run the following commands,
|
|
|
|
|
|
|
|
```bash
|
|
|
|
# Make sure you are in the 'app' directory. Adjust the 'cd' path to
|
|
|
|
# match your set-up.
|
|
|
|
cd ~/www/midpoint/app/
|
|
|
|
|
|
|
|
# If you have created and activated a virtual environment, you can
|
|
|
|
# enter 'python' instead of 'python3'.
|
|
|
|
python3 build-database.py
|
|
|
|
```
|
|
|
|
|
|
|
|
This will clear the database if one is already set-up and then build a
|
|
|
|
new one.
|
|
|
|
|
|
|
|
## Notes on System Limitations
|
|
|
|
|
|
|
|
### 1. Connection Times-Out on Large JSON Packets
|
|
|
|
|
|
|
|
The biggest thing to remember the REST API calls which grab 'all' the
|
|
|
|
light readings from every device, will start to fail fairly quickly
|
|
|
|
after a 'clean install'. The reason why is because the JSON package
|
|
|
|
becomes too large and the connection times out. You might get better
|
|
|
|
results on a more powerfully spec'd server than the one used for the
|
|
|
|
exhibition but I have not tested this so I cannot confirm this.
|
|
|
|
|
|
|
|
### 2. Three Light-Meters and Relays Specified but Only Two (of each) Used
|
|
|
|
|
|
|
|
Ritherdon (the factory) has three welding booths so 'Personal Flash in
|
|
|
|
Real-Time' was designed with this in mind. Over the course of the
|
|
|
|
development process, the use of the third booth was not
|
|
|
|
needed/wanted. By that time, the code, database and models had already
|
|
|
|
been created. I decided to opt for the lazy option and keep the
|
|
|
|
references to the third booth instead of removing it because it was
|
|
|
|
not impeding on the system in any way. I was, also, cautious about the
|
|
|
|
chance of a change happening late into the development cycle were I
|
|
|
|
would need to put it back it.
|