craig.oates
/
overhaul2024
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Code to help with the re-arranging of my life in 2024.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
114 Commits
1 Branch
0 Tags
2.3 MiB
Branch: master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
overhaul2024/roomies-manchester.org

318 lines
16 KiB
Raw Permalink Normal View History Unescape

create roomies-manchester.org file.
3 months ago
#+options: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t
#+options: broken-links:nil c:nil creator:nil d:(not "LOGBOOK") date:t e:t
#+options: email:nil expand-links:t f:t inline:t num:t p:nil pri:nil prop:nil
#+options: stat:t tags:t tasks:t tex:t timestamp:t title:t toc:t todo:t |:t
#+title: Roomies Manchester
#+date: \today
#+author: Craig Oates
#+email: craig@craigoates.net
#+language: en
#+select_tags: export
#+exclude_tags: noexport
#+creator: Emacs 29.1.90 (Org mode 9.7-pre)
#+cite_export:
add processing and summary sections in roomies-manchester.org file.
3 months ago
* Setup Common Lisp Environment
I’ve copied the following code block over from other files. Run it if this is
your first file open in the session.
#+begin_src lisp :session :results silent
(ql:quickload :com.inuoe.jzon) ; JSON parser.
(ql:quickload :dexador) ; HTTP requests.
(ql:quickload :plump) ; HTML/XML parser.
(ql:quickload :lquery) ; HTML/DOM manipulation.
(ql:quickload :lparallel) ; Parallel programming.
(ql:quickload :cl-ppcre) ; RegEx. library.
(ql:quickload :plot/vega) ; Vega plotting library.
(ql:quickload :lisp-stat) ; Stat's library.
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
(ql:quickload :str) ; String library, expands on 'string' library.
#+end_src
create roomies-manchester.org file.
3 months ago
* Gather Roomie Data (Manually)
- [[https://www.roomies.co.uk/][Roomies]]
The website requires you to create an account for full access to the site's
listings. With that said, it does provide enough listings to chart the data. The
amount of listings is tiny, compared to [[file:./spare-room-manchester.org][Spare Room]] (Manc.) and [[file:./right-move-manchester.org][Right Move]]
(Manc.). So, I've downloaded the HTML manually, and going to process that data
instead. The search filters I used:
- Date: 2024-02-29
- Location: Manchester
- Price Range: £0–1200
add processing and summary sections in roomies-manchester.org file.
3 months ago
- Payment Frequency: Month
create roomies-manchester.org file.
3 months ago
There are only twenty-three listings. I’ve stored the HTML is
=raw-data/external/2024-02-29_roomies-manc/=. As is standard behaviour at this
point, I won’t be committing that data to the repositories commit history.
add processing and summary sections in roomies-manchester.org file.
3 months ago
* Clean Up and Parse Data
create roomies-manchester.org file.
3 months ago
add processing and summary sections in roomies-manchester.org file.
3 months ago
I'm applying the same tactic as the previous website reviews. I've separated the
listings into their own files, reducing the chance of attaching the wrong data
to the wrong listings in the CSV files (after parsing the HTML files).
create roomies-manchester.org file.
3 months ago
add processing and summary sections in roomies-manchester.org file.
3 months ago
#+begin_src shell :results silent
mkdir raw-data/external/2024-02-29_roomies-manc-listings/
#+end_src
#+begin_src lisp :results silent
(let ((counter 0))
(loop for file-path
in (directory #P"raw-data/external/2024-02-29_roomies-manc/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listings (lquery:$ doc ".tile" (serialize))))
(loop for item across listings
do (let ((out-path
(merge-pathnames
#P"raw-data/external/2024-02-29_roomies-manc-listings/"
(format nil "listing-~a.html" (write-to-string counter)))))
(with-open-file (out-stream
out-path
:direction :output
:if-exists :supersede)
(format out-stream "~a" item))
(incf counter)))))))
#+end_src
** Create CSV Listings
#+begin_src lisp :results output raw
;; There is a lot of cleaning of text as the function goes along in this
;; one. The best I can say is take your time when making your way through the
;; code. Essentially, the function has to grab the text, convert it into a text
;; format the code can work with and then strip/replace tokens within the text
;; to make it readable in the CSV file. Yes, this is not my best work, but it
;; works and I don't intend to run this in production for years on end. Get the
;; data, parse it and get out.
(let ((filepath #P"working-data/2024-02-29-roomies-manc.csv"))
(with-open-file (out-stream
filepath
:direction :output
:if-exists :supersede)
(let ((row-id 0))
(format out-stream "ROW-ID,LOCATION,RENT,BILLS-INC,URL~%")
(loop for file-path
in (directory #P"raw-data/external/2024-02-29_roomies-manc-listings/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listing (lquery:$ doc ".tile" (text)))
(price (lquery:$ doc ".text-white" (text)))
(cleaned-price
(cl-ppcre:regex-replace-all
"[\\s\\n]+"
(format nil
(str:replace-all
"," ""
(format nil "~s" (aref price 0))))
" "))
(just-price (format nil "~a"
(first (cl-ppcre:all-matches-as-strings
"\\d+" cleaned-price))))
(bills (format nil "~a"
(first (cl-ppcre:all-matches-as-strings
"inc." cleaned-price))))
(location (lquery:$ doc ".text-lg" (text)))
(cleaned-loc
(cl-ppcre:regex-replace-all
"[\\s\\n]+" (format nil "~a" (aref location 0)) " "))
(link (lquery:$ doc "a" (attr :href))))
(format out-stream "~d,~a,~d,~a,~a~%"
row-id
(string-trim " "
(str:replace-all "," " " cleaned-loc))
just-price
(if (string= "inc." bills)
(format nil "Yes")
(format nil "No"))
(aref link 0)))
(incf row-id)))))
(format t "[[file:./~a]]" filepath))
create roomies-manchester.org file.
3 months ago
#+end_src
add processing and summary sections in roomies-manchester.org file.
3 months ago
#+RESULTS:
[[file:./working-data/2024-02-29-roomies-manc.csv]]
The code above produced a CSV file with a few ~NIL~ results in the ~RENT~
column. These listings are either 'featured' or a 'verified account'. Because
the numbers are so small, I've updated them manually.
* Explore CSV Data for Roomies (2024-02-29)
#+begin_src lisp :session
(lisp-stat:defdf *room-manc*
(lisp-stat:read-csv #P"working-data/2024-02-29-roomies-manc.csv"))
#+end_src
#+RESULTS:
: #<DATA-FRAME:DATA-FRAME (24 observations of 5 variables)>
Because the number of listing are so small, compared to [[file:./right-move-manchester.org][Right Move]] and [[file:./spare-room-manchester.org][Spare
Room]] (both Manc.) listings, I can output the data here.
#+begin_src lisp :session :results output drawer
(lisp-stat:print-markdown *room-manc*)
#+end_src
#+RESULTS:
:results:
| ROW-ID | LOCATION | RENT | BILLS-INC | URL |
|--------+------------------------------------------+------+-----------+----------------------------------------|
| 0 | Nobel Way Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/502205 |
| 1 | Audenshaw England | 700 | No | https://www.roomies.co.uk/rooms/504567 |
| 2 | Openshaw Walk Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/392849 |
| 3 | Manchester Road Denton England | 400 | Yes | https://www.roomies.co.uk/rooms/484296 |
| 4 | Manchester Street Old Trafford England | 750 | No | https://www.roomies.co.uk/rooms/484350 |
| 5 | Beswick Street Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/481551 |
| 6 | Mealhouse Lane Atherton England | 650 | No | https://www.roomies.co.uk/rooms/476315 |
| 7 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/476586 |
| 8 | Old Trafford England | 600 | Yes | https://www.roomies.co.uk/rooms/439451 |
| 9 | Droylsden Road Manchester England | 550 | No | https://www.roomies.co.uk/rooms/457342 |
| 10 | Audenshaw England | 550 | No | https://www.roomies.co.uk/rooms/453458 |
| 11 | Briarfield Road Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/328052 |
| 12 | Manchester England | 700 | No | https://www.roomies.co.uk/rooms/472435 |
| 13 | Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/406205 |
| 14 | Manchester England | 580 | Yes | https://www.roomies.co.uk/rooms/434137 |
| 15 | Yates Street Rhodes England | 430 | Yes | https://www.roomies.co.uk/rooms/393284 |
| 16 | Stockdale Road Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/344733 |
| 17 | Manchester England | 800 | No | https://www.roomies.co.uk/rooms/472436 |
| 18 | Talbot Road Manchester England | 450 | Yes | https://www.roomies.co.uk/rooms/267267 |
| 19 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/396587 |
| 20 | Chelsfield Grove Manchester England | 550 | No | https://www.roomies.co.uk/rooms/499082 |
| 21 | Manchester England | 475 | Yes | https://www.roomies.co.uk/rooms/361541 |
| 22 | Manchester England | 575 | Yes | https://www.roomies.co.uk/rooms/354296 |
| 23 | Manchester England | 650 | Yes | https://www.roomies.co.uk/rooms/489135 |
:end:
I've assumed the position of excluding the listings which don't include bills in
the rent advertised. So, I'm need to create a new data-frame, which filters out
the listings which don't include bills.
#+begin_src lisp :session
(lisp-stat:defdf *room-manc-filt*
(lisp-stat:filter-rows *room-manc* '(string= "Yes" bills-inc)))
#+end_src
#+RESULTS:
: #<DATA-FRAME:DATA-FRAME (16 observations of 5 variables)>
The data with just the listings which include bills in the rent.
#+begin_src lisp :session :results output drawer
(lisp-stat:print-markdown *room-manc-filt*)
#+end_src
#+RESULTS:
:results:
| ROW-ID | LOCATION | RENT | BILLS-INC | URL |
|--------+--------------------------------------+------+-----------+----------------------------------------|
| 0 | Nobel Way Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/502205 |
| 2 | Openshaw Walk Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/392849 |
| 3 | Manchester Road Denton England | 400 | Yes | https://www.roomies.co.uk/rooms/484296 |
| 5 | Beswick Street Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/481551 |
| 7 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/476586 |
| 8 | Old Trafford England | 600 | Yes | https://www.roomies.co.uk/rooms/439451 |
| 11 | Briarfield Road Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/328052 |
| 13 | Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/406205 |
| 14 | Manchester England | 580 | Yes | https://www.roomies.co.uk/rooms/434137 |
| 15 | Yates Street Rhodes England | 430 | Yes | https://www.roomies.co.uk/rooms/393284 |
| 16 | Stockdale Road Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/344733 |
| 18 | Talbot Road Manchester England | 450 | Yes | https://www.roomies.co.uk/rooms/267267 |
| 19 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/396587 |
| 21 | Manchester England | 475 | Yes | https://www.roomies.co.uk/rooms/361541 |
| 22 | Manchester England | 575 | Yes | https://www.roomies.co.uk/rooms/354296 |
| 23 | Manchester England | 650 | Yes | https://www.roomies.co.uk/rooms/489135 |
:end:
#+begin_src lisp :session :results file
(vega:defplot roomie-monthly-manc
`(:title "Rent Rates for Manchester on Roomies (29/02/2024)"
:width 600
:height 600
:data ,*room-manc-filt*
:layer #((:mark (:type :bar)
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0))
:y (:field :rent :title "Monthly Rent with Bills Inc. (£)" :type :quantitative)
:tooltip (:field :rent)))
(:mark (:type rule :color "darkorange" :size 3)
:encoding (:y (:field :rent :type :quantitative :aggregate :average)
:tooltip (:field :rent :type :quantitative :aggregate :average))))))
(vega:write-html roomie-monthly-manc "renders/2024-02-29-roomies-rent-manc.html")
#+end_src
#+RESULTS:
[[file:renders/2024-02-29-roomies-rent-manc.html]]
[[file:./renders/2024-02-29-roomies-rent-manc.png]]
#+begin_src lisp :session :results output code raw
(format t "- Mean Rent: £ ~a~%" (float (lisp-stat:mean *room-manc-filt*:rent)))
(format t "- Min. Rent: £ ~d~%" (reduce #'min *room-manc-filt*:rent))
(format t "- Max. Rent: £ ~d" (reduce #'max *room-manc-filt*:rent))
#+end_src
#+RESULTS:
- Mean Rent: £ 641.25
- Min. Rent: £ 400
- Max. Rent: £ 1000
* Summary of Roomies Data
#+begin_src calc :results output
641.25 * 12
#+end_src
#+RESULTS:
: 7695
Based on the average rent (inc. bills) for Roomies, I would need to make around
£8,000/yr. to cover my living expenses. This does not include, travel, food,
clothing, socialising or income tax payments. In the other Manchester reviews,
I've been adding £5,000 on top of the basic rent total. This is so I have a
minimum threshold for money I need to make to cover my costs (i.e. to
survive). So, adding the £5,000 on top…
#+begin_src calc :results output
7695 + 5000
#+end_src
#+RESULTS:
: 12695
Using £12695 as the starting point (see [[file:./uk-wage-tax.org][UK Wage and Tax Rates]]),
#+begin_src lisp :results output raw
(let* ((earning-target 12695)
(p-allow 12570)
(taxable-income (- earning-target p-allow))
correct Income Tax calculation error in roomies-manchester.org file.
3 months ago
(tax-to-pay (* taxable-income 0.2))
add processing and summary sections in roomies-manchester.org file.
3 months ago
(total (- earning-target tax-to-pay)))
(format t "- Annual Target Salary: £~a~%" earning-target)
(format t "- Part of Salary which is Taxable: £~a~%" taxable-income)
(format t "- Tax to Pay: £~a~%" tax-to-pay)
(format t "- Salary After Tax: £~a~%" total))
#+end_src
#+RESULTS:
- Annual Target Salary: £12695
- Part of Salary which is Taxable: £125
correct Income Tax calculation error in roomies-manchester.org file.
3 months ago
- Tax to Pay: £25.0
- Salary After Tax: £12670.0
add processing and summary sections in roomies-manchester.org file.
3 months ago
| Time Span | Value After Tax (£) | Mean Rent (£) |
|-----------------------+---------------------+---------------|
correct Income Tax calculation error in roomies-manchester.org file.
3 months ago
| Annually | 12670 | 641.25 |
| Monthly (Before Rent) | 1055.8333 | |
| Monthly (After Rent) | 414.5833 | |
| Weekly (After Rent) | 103.64583 | |
| Daily (After Rent) | 14.806547 | |
add processing and summary sections in roomies-manchester.org file.
3 months ago
#+TBLFM: @3$2=@-1/12::@4$2=@-1-@-2$+1::@5$2=@-1/4::@6$2=@-1/7
create roomies-manchester.org file.
3 months ago
correct Income Tax calculation error in roomies-manchester.org file.
3 months ago
The spending limit of £14.80/day falls inline with the other Manchester-based
add processing and summary sections in roomies-manchester.org file.
3 months ago
files. Not great but these figures represent the minimum I need to aim for,
regarding wages/salaries.