@ -12,6 +12,24 @@
#+creator : Emacs 29.1.90 (Org mode 9.7-pre)
#+cite_export :
* Setup Common Lisp Environment
I’ve copied the following code block over from other files. Run it if this is
your first file open in the session.
#+begin_src lisp :session :results silent
(ql:quickload :com.inuoe.jzon) ; JSON parser.
(ql:quickload :dexador) ; HTTP requests.
(ql:quickload :plump) ; HTML/XML parser.
(ql:quickload :lquery) ; HTML/DOM manipulation.
(ql:quickload :lparallel) ; Parallel programming.
(ql:quickload :cl-ppcre) ; RegEx. library.
(ql:quickload :plot/vega) ; Vega plotting library.
(ql:quickload :lisp-stat) ; Stat's library.
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
(ql:quickload :str) ; String library, expands on 'string' library.
#+end_src
* Gather Roomie Data (Manually)
- [[https://www.roomies.co.uk/ ][Roomies ]]
@ -25,29 +43,275 @@ instead. The search filters I used:
- Date: 2024-02-29
- Location: Manchester
- Price Range: £0–1200
- Payment Frequency: Month
There are only twenty-three listings. I’ve stored the HTML is
=raw-data/external/2024-02-29_roomies-manc/= . As is standard behaviour at this
point, I won’t be committing that data to the repositories commit history.
* Setup Common Lisp Environment
* Clean Up and Parse Data
I’ve copied the following code block over from other files. Run it if this is
your first file open in the session.
I'm applying the same tactic as the previous website reviews. I've separated the
listings into their own files, reducing the chance of attaching the wrong data
to the wrong listings in the CSV files (after parsing the HTML files).
#+begin_src lisp :session :results silent
(ql:quickload :com.inuoe.jzon) ; JSON parser.
(ql:quickload :dexador) ; HTTP requests.
(ql:quickload :plump) ; HTML/XML parser.
(ql:quickload :lquery) ; HTML/DOM manipulation.
(ql:quickload :lparallel) ; Parallel programming.
(ql:quickload :cl-ppcre) ; RegEx. library.
(ql:quickload :plot/vega) ; Vega plotting library.
(ql:quickload :lisp-stat) ; Stat's library.
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
(ql:quickload :str) ; String library, expands on 'string' library.
#+begin_src shell :results silent
mkdir raw-data/external/2024-02-29_roomies-manc-listings/
#+end_src
#+begin_src lisp :results silent
(let ((counter 0))
(loop for file-path
in (directory #P"raw-data/external/2024-02-29_roomies-manc/ *.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listings (lquery:$ doc ".tile" (serialize))))
(loop for item across listings
do (let ((out-path
(merge-pathnames
#P"raw-data/external/2024-02-29_roomies-manc-listings/ "
(format nil "listing-~a.html" (write-to-string counter)))))
(with-open-file (out-stream
out-path
:direction :output
:if-exists :supersede)
(format out-stream "~a" item))
(incf counter)))))))
#+end_src
** Create CSV Listings
#+begin_src lisp :results output raw
;; There is a lot of cleaning of text as the function goes along in this
;; one. The best I can say is take your time when making your way through the
;; code. Essentially, the function has to grab the text, convert it into a text
;; format the code can work with and then strip/replace tokens within the text
;; to make it readable in the CSV file. Yes, this is not my best work, but it
;; works and I don't intend to run this in production for years on end. Get the
;; data, parse it and get out.
(let ((filepath #P"working-data/2024-02-29-roomies-manc.csv"))
(with-open-file (out-stream
filepath
:direction :output
:if-exists :supersede)
(let ((row-id 0))
(format out-stream "ROW-ID,LOCATION,RENT,BILLS-INC,URL~%")
(loop for file-path
in (directory #P"raw-data/external/2024-02-29_roomies-manc-listings/ *.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listing (lquery:$ doc ".tile" (text)))
(price (lquery:$ doc ".text-white" (text)))
(cleaned-price
(cl-ppcre:regex-replace-all
"[\\s\\n]+"
(format nil
(str:replace-all
"," ""
(format nil "~s" (aref price 0))))
" "))
(just-price (format nil "~a"
(first (cl-ppcre:all-matches-as-strings
"\\d+" cleaned-price))))
(bills (format nil "~a"
(first (cl-ppcre:all-matches-as-strings
"inc." cleaned-price))))
(location (lquery:$ doc ".text-lg" (text)))
(cleaned-loc
(cl-ppcre:regex-replace-all
"[\\s\\n]+" (format nil "~a" (aref location 0)) " "))
(link (lquery:$ doc "a" (attr :href))))
(format out-stream "~d,~a,~d,~a,~a~ %"
row-id
(string-trim " "
(str:replace-all "," " " cleaned-loc))
just-price
(if (string= "inc." bills)
(format nil "Yes")
(format nil "No"))
(aref link 0)))
(incf row-id)))))
(format t "[[file:./~a ]]" filepath))
#+end_src
* TODO Clean Up and Parse Data
#+RESULTS :
[[file:./working-data/2024-02-29-roomies-manc.csv ]]
The code above produced a CSV file with a few ~NIL~ results in the ~RENT~
column. These listings are either 'featured' or a 'verified account'. Because
the numbers are so small, I've updated them manually.
* Explore CSV Data for Roomies (2024-02-29)
#+begin_src lisp :session
(lisp-stat:defdf *room-manc*
(lisp-stat:read-csv #P"working-data/2024-02-29-roomies-manc.csv"))
#+end_src
#+RESULTS :
: #<DATA-FRAME:DATA-FRAME (24 observations of 5 variables) >
Because the number of listing are so small, compared to [[file:./right-move-manchester.org ][Right Move ]] and [[file:./spare-room-manchester.org][Spare
Room]] (both Manc.) listings, I can output the data here.
#+begin_src lisp :session :results output drawer
(lisp-stat:print-markdown *room-manc* )
#+end_src
#+RESULTS :
:results:
| ROW-ID | LOCATION | RENT | BILLS-INC | URL |
|--------+------------------------------------------+------+-----------+----------------------------------------|
| 0 | Nobel Way Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/502205 |
| 1 | Audenshaw England | 700 | No | https://www.roomies.co.uk/rooms/504567 |
| 2 | Openshaw Walk Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/392849 |
| 3 | Manchester Road Denton England | 400 | Yes | https://www.roomies.co.uk/rooms/484296 |
| 4 | Manchester Street Old Trafford England | 750 | No | https://www.roomies.co.uk/rooms/484350 |
| 5 | Beswick Street Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/481551 |
| 6 | Mealhouse Lane Atherton England | 650 | No | https://www.roomies.co.uk/rooms/476315 |
| 7 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/476586 |
| 8 | Old Trafford England | 600 | Yes | https://www.roomies.co.uk/rooms/439451 |
| 9 | Droylsden Road Manchester England | 550 | No | https://www.roomies.co.uk/rooms/457342 |
| 10 | Audenshaw England | 550 | No | https://www.roomies.co.uk/rooms/453458 |
| 11 | Briarfield Road Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/328052 |
| 12 | Manchester England | 700 | No | https://www.roomies.co.uk/rooms/472435 |
| 13 | Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/406205 |
| 14 | Manchester England | 580 | Yes | https://www.roomies.co.uk/rooms/434137 |
| 15 | Yates Street Rhodes England | 430 | Yes | https://www.roomies.co.uk/rooms/393284 |
| 16 | Stockdale Road Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/344733 |
| 17 | Manchester England | 800 | No | https://www.roomies.co.uk/rooms/472436 |
| 18 | Talbot Road Manchester England | 450 | Yes | https://www.roomies.co.uk/rooms/267267 |
| 19 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/396587 |
| 20 | Chelsfield Grove Manchester England | 550 | No | https://www.roomies.co.uk/rooms/499082 |
| 21 | Manchester England | 475 | Yes | https://www.roomies.co.uk/rooms/361541 |
| 22 | Manchester England | 575 | Yes | https://www.roomies.co.uk/rooms/354296 |
| 23 | Manchester England | 650 | Yes | https://www.roomies.co.uk/rooms/489135 |
:end:
I've assumed the position of excluding the listings which don't include bills in
the rent advertised. So, I'm need to create a new data-frame, which filters out
the listings which don't include bills.
#+begin_src lisp :session
(lisp-stat:defdf *room-manc-filt*
(lisp-stat:filter-rows *room-manc* '(string= "Yes" bills-inc)))
#+end_src
#+RESULTS :
: #<DATA-FRAME:DATA-FRAME (16 observations of 5 variables) >
The data with just the listings which include bills in the rent.
#+begin_src lisp :session :results output drawer
(lisp-stat:print-markdown *room-manc-filt* )
#+end_src
#+RESULTS :
:results:
| ROW-ID | LOCATION | RENT | BILLS-INC | URL |
|--------+--------------------------------------+------+-----------+----------------------------------------|
| 0 | Nobel Way Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/502205 |
| 2 | Openshaw Walk Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/392849 |
| 3 | Manchester Road Denton England | 400 | Yes | https://www.roomies.co.uk/rooms/484296 |
| 5 | Beswick Street Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/481551 |
| 7 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/476586 |
| 8 | Old Trafford England | 600 | Yes | https://www.roomies.co.uk/rooms/439451 |
| 11 | Briarfield Road Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/328052 |
| 13 | Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/406205 |
| 14 | Manchester England | 580 | Yes | https://www.roomies.co.uk/rooms/434137 |
| 15 | Yates Street Rhodes England | 430 | Yes | https://www.roomies.co.uk/rooms/393284 |
| 16 | Stockdale Road Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/344733 |
| 18 | Talbot Road Manchester England | 450 | Yes | https://www.roomies.co.uk/rooms/267267 |
| 19 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/396587 |
| 21 | Manchester England | 475 | Yes | https://www.roomies.co.uk/rooms/361541 |
| 22 | Manchester England | 575 | Yes | https://www.roomies.co.uk/rooms/354296 |
| 23 | Manchester England | 650 | Yes | https://www.roomies.co.uk/rooms/489135 |
:end:
#+begin_src lisp :session :results file
(vega:defplot roomie-monthly-manc
`(:title "Rent Rates for Manchester on Roomies (29/02/2024)"
:width 600
:height 600
:data ,*room-manc-filt*
:layer #((:mark (:type :bar)
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0))
:y (:field :rent :title "Monthly Rent with Bills Inc. (£)" :type :quantitative)
:tooltip (:field :rent)))
(:mark (:type rule :color "darkorange" :size 3)
:encoding (:y (:field :rent :type :quantitative :aggregate :average)
:tooltip (:field :rent :type :quantitative :aggregate :average))))))
(vega:write-html roomie-monthly-manc "renders/2024-02-29-roomies-rent-manc.html")
#+end_src
#+RESULTS :
[[file:renders/2024-02-29-roomies-rent-manc.html ]]
[[file:./renders/2024-02-29-roomies-rent-manc.png ]]
#+begin_src lisp :session :results output code raw
(format t "- Mean Rent: £ ~a~ %" (float (lisp-stat:mean *room-manc-filt* :rent)))
(format t "- Min. Rent: £ ~d~ %" (reduce #'min *room-manc-filt* :rent))
(format t "- Max. Rent: £ ~d" (reduce #'max *room-manc-filt* :rent))
#+end_src
#+RESULTS :
- Mean Rent: £ 641.25
- Min. Rent: £ 400
- Max. Rent: £ 1000
* Summary of Roomies Data
#+begin_src calc :results output
641.25 * 12
#+end_src
#+RESULTS :
: 7695
Based on the average rent (inc. bills) for Roomies, I would need to make around
£8,000/yr. to cover my living expenses. This does not include, travel, food,
clothing, socialising or income tax payments. In the other Manchester reviews,
I've been adding £5,000 on top of the basic rent total. This is so I have a
minimum threshold for money I need to make to cover my costs (i.e. to
survive). So, adding the £5,000 on top…
#+begin_src calc :results output
7695 + 5000
#+end_src
#+RESULTS :
: 12695
Using £12695 as the starting point (see [[file:./uk-wage-tax.org ][UK Wage and Tax Rates ]]),
#+begin_src lisp :results output raw
(let* ((earning-target 12695)
(p-allow 12570)
(taxable-income (- earning-target p-allow))
(tax-to-pay (* taxable-income 0.8))
(total (- earning-target tax-to-pay)))
(format t "- Annual Target Salary: £~a~ %" earning-target)
(format t "- Part of Salary which is Taxable: £~a~ %" taxable-income)
(format t "- Tax to Pay: £~a~ %" tax-to-pay)
(format t "- Salary After Tax: £~a~ %" total))
#+end_src
#+RESULTS :
- Annual Target Salary: £12695
- Part of Salary which is Taxable: £125
- Tax to Pay: £100.0
- Salary After Tax: £12595.0
| Time Span | Value After Tax (£) | Mean Rent (£) |
|-----------------------+---------------------+---------------|
| Annually | 12595 | 641.25 |
| Monthly (Before Rent) | 1049.5833 | |
| Monthly (After Rent) | 408.3333 | |
| Weekly (After Rent) | 102.08333 | |
| Daily (After Rent) | 14.583333 | |
#+TBLFM : @3$2=@-1/12::@4$2=@-1-@-2$+1::@5$2=@-1/4::@6$2=@-1/7
This bit I need to start work on.
The spending limit of £14.58/day falls inline with the other Manchester-based
files. Not great but these figures represent the minimum I need to aim for,
regarding wages/salaries.
Craig Oates
