16 KiB
Roomies Manchester
- Setup Common Lisp Environment
- Gather Roomie Data (Manually)
- Clean Up and Parse Data
- Explore CSV Data for Roomies (2024-02-29)
- Summary of Roomies Data
Setup Common Lisp Environment
I’ve copied the following code block over from other files. Run it if this is your first file open in the session.
(ql:quickload :com.inuoe.jzon) ; JSON parser.
(ql:quickload :dexador) ; HTTP requests.
(ql:quickload :plump) ; HTML/XML parser.
(ql:quickload :lquery) ; HTML/DOM manipulation.
(ql:quickload :lparallel) ; Parallel programming.
(ql:quickload :cl-ppcre) ; RegEx. library.
(ql:quickload :plot/vega) ; Vega plotting library.
(ql:quickload :lisp-stat) ; Stat's library.
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
(ql:quickload :str) ; String library, expands on 'string' library.Gather Roomie Data (Manually)
The website requires you to create an account for full access to the site's listings. With that said, it does provide enough listings to chart the data. The amount of listings is tiny, compared to Spare Room (Manc.) and Right Move (Manc.). So, I've downloaded the HTML manually, and going to process that data instead. The search filters I used:
- Date: 2024-02-29
- Location: Manchester
- Price Range: £0–1200
- Payment Frequency: Month
There are only twenty-three listings. I’ve stored the HTML is
raw-data/external/2024-02-29_roomies-manc/. As is standard behaviour at this
point, I won’t be committing that data to the repositories commit history.
Clean Up and Parse Data
I'm applying the same tactic as the previous website reviews. I've separated the listings into their own files, reducing the chance of attaching the wrong data to the wrong listings in the CSV files (after parsing the HTML files).
mkdir raw-data/external/2024-02-29_roomies-manc-listings/ (let ((counter 0))
(loop for file-path
in (directory #P"raw-data/external/2024-02-29_roomies-manc/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listings (lquery:$ doc ".tile" (serialize))))
(loop for item across listings
do (let ((out-path
(merge-pathnames
#P"raw-data/external/2024-02-29_roomies-manc-listings/"
(format nil "listing-~a.html" (write-to-string counter)))))
(with-open-file (out-stream
out-path
:direction :output
:if-exists :supersede)
(format out-stream "~a" item))
(incf counter)))))))Create CSV Listings
;; There is a lot of cleaning of text as the function goes along in this
;; one. The best I can say is take your time when making your way through the
;; code. Essentially, the function has to grab the text, convert it into a text
;; format the code can work with and then strip/replace tokens within the text
;; to make it readable in the CSV file. Yes, this is not my best work, but it
;; works and I don't intend to run this in production for years on end. Get the
;; data, parse it and get out.
(let ((filepath #P"working-data/2024-02-29-roomies-manc.csv"))
(with-open-file (out-stream
filepath
:direction :output
:if-exists :supersede)
(let ((row-id 0))
(format out-stream "ROW-ID,LOCATION,RENT,BILLS-INC,URL~%")
(loop for file-path
in (directory #P"raw-data/external/2024-02-29_roomies-manc-listings/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listing (lquery:$ doc ".tile" (text)))
(price (lquery:$ doc ".text-white" (text)))
(cleaned-price
(cl-ppcre:regex-replace-all
"[\\s\\n]+"
(format nil
(str:replace-all
"," ""
(format nil "~s" (aref price 0))))
" "))
(just-price (format nil "~a"
(first (cl-ppcre:all-matches-as-strings
"\\d+" cleaned-price))))
(bills (format nil "~a"
(first (cl-ppcre:all-matches-as-strings
"inc." cleaned-price))))
(location (lquery:$ doc ".text-lg" (text)))
(cleaned-loc
(cl-ppcre:regex-replace-all
"[\\s\\n]+" (format nil "~a" (aref location 0)) " "))
(link (lquery:$ doc "a" (attr :href))))
(format out-stream "~d,~a,~d,~a,~a~%"
row-id
(string-trim " "
(str:replace-all "," " " cleaned-loc))
just-price
(if (string= "inc." bills)
(format nil "Yes")
(format nil "No"))
(aref link 0)))
(incf row-id)))))
(format t "[[file:./~a]]" filepath))/craig.oates/overhaul2024/src/branch/master/working-data/2024-02-29-roomies-manc.csv
The code above produced a CSV file with a few NIL results in the RENT
column. These listings are either 'featured' or a 'verified account'. Because
the numbers are so small, I've updated them manually.
Explore CSV Data for Roomies (2024-02-29)
(lisp-stat:defdf *room-manc*
(lisp-stat:read-csv #P"working-data/2024-02-29-roomies-manc.csv"))#<DATA-FRAME:DATA-FRAME (24 observations of 5 variables)>
Because the number of listing are so small, compared to Right Move and Spare Room (both Manc.) listings, I can output the data here.
(lisp-stat:print-markdown *room-manc*)| ROW-ID | LOCATION | RENT | BILLS-INC | URL |
|---|---|---|---|---|
| 0 | Nobel Way Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/502205 |
| 1 | Audenshaw England | 700 | No | https://www.roomies.co.uk/rooms/504567 |
| 2 | Openshaw Walk Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/392849 |
| 3 | Manchester Road Denton England | 400 | Yes | https://www.roomies.co.uk/rooms/484296 |
| 4 | Manchester Street Old Trafford England | 750 | No | https://www.roomies.co.uk/rooms/484350 |
| 5 | Beswick Street Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/481551 |
| 6 | Mealhouse Lane Atherton England | 650 | No | https://www.roomies.co.uk/rooms/476315 |
| 7 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/476586 |
| 8 | Old Trafford England | 600 | Yes | https://www.roomies.co.uk/rooms/439451 |
| 9 | Droylsden Road Manchester England | 550 | No | https://www.roomies.co.uk/rooms/457342 |
| 10 | Audenshaw England | 550 | No | https://www.roomies.co.uk/rooms/453458 |
| 11 | Briarfield Road Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/328052 |
| 12 | Manchester England | 700 | No | https://www.roomies.co.uk/rooms/472435 |
| 13 | Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/406205 |
| 14 | Manchester England | 580 | Yes | https://www.roomies.co.uk/rooms/434137 |
| 15 | Yates Street Rhodes England | 430 | Yes | https://www.roomies.co.uk/rooms/393284 |
| 16 | Stockdale Road Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/344733 |
| 17 | Manchester England | 800 | No | https://www.roomies.co.uk/rooms/472436 |
| 18 | Talbot Road Manchester England | 450 | Yes | https://www.roomies.co.uk/rooms/267267 |
| 19 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/396587 |
| 20 | Chelsfield Grove Manchester England | 550 | No | https://www.roomies.co.uk/rooms/499082 |
| 21 | Manchester England | 475 | Yes | https://www.roomies.co.uk/rooms/361541 |
| 22 | Manchester England | 575 | Yes | https://www.roomies.co.uk/rooms/354296 |
| 23 | Manchester England | 650 | Yes | https://www.roomies.co.uk/rooms/489135 |
I've assumed the position of excluding the listings which don't include bills in the rent advertised. So, I'm need to create a new data-frame, which filters out the listings which don't include bills.
(lisp-stat:defdf *room-manc-filt*
(lisp-stat:filter-rows *room-manc* '(string= "Yes" bills-inc)))#<DATA-FRAME:DATA-FRAME (16 observations of 5 variables)>
The data with just the listings which include bills in the rent.
(lisp-stat:print-markdown *room-manc-filt*)| ROW-ID | LOCATION | RENT | BILLS-INC | URL |
|---|---|---|---|---|
| 0 | Nobel Way Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/502205 |
| 2 | Openshaw Walk Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/392849 |
| 3 | Manchester Road Denton England | 400 | Yes | https://www.roomies.co.uk/rooms/484296 |
| 5 | Beswick Street Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/481551 |
| 7 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/476586 |
| 8 | Old Trafford England | 600 | Yes | https://www.roomies.co.uk/rooms/439451 |
| 11 | Briarfield Road Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/328052 |
| 13 | Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/406205 |
| 14 | Manchester England | 580 | Yes | https://www.roomies.co.uk/rooms/434137 |
| 15 | Yates Street Rhodes England | 430 | Yes | https://www.roomies.co.uk/rooms/393284 |
| 16 | Stockdale Road Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/344733 |
| 18 | Talbot Road Manchester England | 450 | Yes | https://www.roomies.co.uk/rooms/267267 |
| 19 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/396587 |
| 21 | Manchester England | 475 | Yes | https://www.roomies.co.uk/rooms/361541 |
| 22 | Manchester England | 575 | Yes | https://www.roomies.co.uk/rooms/354296 |
| 23 | Manchester England | 650 | Yes | https://www.roomies.co.uk/rooms/489135 |
(vega:defplot roomie-monthly-manc
`(:title "Rent Rates for Manchester on Roomies (29/02/2024)"
:width 600
:height 600
:data ,*room-manc-filt*
:layer #((:mark (:type :bar)
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0))
:y (:field :rent :title "Monthly Rent with Bills Inc. (£)" :type :quantitative)
:tooltip (:field :rent)))
(:mark (:type rule :color "darkorange" :size 3)
:encoding (:y (:field :rent :type :quantitative :aggregate :average)
:tooltip (:field :rent :type :quantitative :aggregate :average))))))
(vega:write-html roomie-monthly-manc "renders/2024-02-29-roomies-rent-manc.html")/craig.oates/overhaul2024/src/branch/master/renders/2024-02-29-roomies-rent-manc.html
(format t "- Mean Rent: £ ~a~%" (float (lisp-stat:mean *room-manc-filt*:rent)))
(format t "- Min. Rent: £ ~d~%" (reduce #'min *room-manc-filt*:rent))
(format t "- Max. Rent: £ ~d" (reduce #'max *room-manc-filt*:rent))- Mean Rent: £ 641.25
- Min. Rent: £ 400
- Max. Rent: £ 1000
Summary of Roomies Data
641.25 * 127695
Based on the average rent (inc. bills) for Roomies, I would need to make around £8,000/yr. to cover my living expenses. This does not include, travel, food, clothing, socialising or income tax payments. In the other Manchester reviews, I've been adding £5,000 on top of the basic rent total. This is so I have a minimum threshold for money I need to make to cover my costs (i.e. to survive). So, adding the £5,000 on top…
7695 + 500012695
Using £12695 as the starting point (see UK Wage and Tax Rates),
(let* ((earning-target 12695)
(p-allow 12570)
(taxable-income (- earning-target p-allow))
(tax-to-pay (* taxable-income 0.2))
(total (- earning-target tax-to-pay)))
(format t "- Annual Target Salary: £~a~%" earning-target)
(format t "- Part of Salary which is Taxable: £~a~%" taxable-income)
(format t "- Tax to Pay: £~a~%" tax-to-pay)
(format t "- Salary After Tax: £~a~%" total))- Annual Target Salary: £12695
- Part of Salary which is Taxable: £125
- Tax to Pay: £25.0
- Salary After Tax: £12670.0
| Time Span | Value After Tax (£) | Mean Rent (£) |
|---|---|---|
| Annually | 12670 | 641.25 |
| Monthly (Before Rent) | 1055.8333 | |
| Monthly (After Rent) | 414.5833 | |
| Weekly (After Rent) | 103.64583 | |
| Daily (After Rent) | 14.806547 |
The spending limit of £14.80/day falls inline with the other Manchester-based files. Not great but these figures represent the minimum I need to aim for, regarding wages/salaries.