14 KiB
Ideal Flatmate Manchester
- Gather Ideal Flatmate Data (Manually)
- Setup Common Lisp Environment
- Clean Up and Parse Data
- Explore CSV Data for Ideal Flatmate (2024-02-24)
- Summary of Ideal Flatmate Data
Gather Ideal Flatmate Data (Manually)
Having had a quick look on the website and did a search with the following filters:
- Date: 2024-02-24 Sat
- Location: Manchester (City)
- Price Range: £0-1200
- Distance: +20 km
There are only two pages of results – sixteen listings in total. So, I’ve just
saved the HTML manually, from within the browser. Because they are HTML files
and come with JavaScript, CSS, images etc., I stored in
raw-data/external/2024-02-24_ideal-flatmate-manc. These files will not be
committed to the repository because I don’t want to clog it up with excess files
and images. I just want the rent rates and location data.
Setup Common Lisp Environment
You will not need to execute this code block if you've already set up SLIME in another ORG file. This is just in case this is the only file you're working on today, or it's your first file of the day.
Run m-x slime before running the following code. And, make note of the
:session attribute. It allows you to use the code in the code block to be use
in other code blocks which also use the :session attribute.
(ql:quickload :com.inuoe.jzon) ; JSON parser.
(ql:quickload :dexador) ; HTTP requests.
(ql:quickload :plump) ; HTML/XML parser.
(ql:quickload :lquery) ; HTML/DOM manipulation.
(ql:quickload :lparallel) ; Parallel programming.
(ql:quickload :cl-ppcre) ; RegEx. library.
(ql:quickload :plot/vega) ; Vega plotting library.
(ql:quickload :lisp-stat) ; Stat's library.
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
(ql:quickload :str) ; String library, expands on 'string' library.Clean Up and Parse Data
I'm taking a leaf out of the Spare Room (Manc) book and separating the individual listings into their own files. I've already got code I can quickly adapt to do this and it gives me more confidence around attaching values to the wrong listings.
mkdir raw-data/external/2024-02-24_ideal-flatmate-manc-listings/ (let ((counter 0))
(loop for file-path
in (directory #P"raw-data/external/2024-02-24_ideal-flatmate-manc/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listings (lquery:$ doc ".card-infos-flex-row" (serialize))))
(loop for item across listings
do (let ((out-path
(merge-pathnames #P"raw-data/external/2024-02-24_ideal-flatmate-manc-listings/"
(format nil "listing-~a.html" (write-to-string counter)))))
(with-open-file (out-stream
out-path
:direction :output
:if-exists :supersede)
(format out-stream "~a" item))
(incf counter)))))))Create CSV of Listings
(let ((filepath #P"working-data/2024-02-24-ideal-flatmate-manc.csv"))
(with-open-file (out-stream
filepath
:direction :output
:if-exists :supersede)
(let ((row-id 0))
(format out-stream "ROW-ID,RAW-LISTING-INFO,RENT,LOCATION,URL~%")
(loop for file-path
in (directory #P"raw-data/external/2024-02-24_ideal-flatmate-manc-listings/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listing (lquery:$ doc ".card-infos-left" (text)))
(cleaned-price
(first (cl-ppcre:all-matches-as-strings
"\\d+"
(first (str:split "room" (aref listing 0))))))
(cleaned-location
(str:replace-all "," ""
(second (str:split "room" (aref listing 0)))))
(url (lquery:$ doc "a" (attr "href"))))
(format out-stream "~d,~s,~d,~a,~a~%"
row-id
(aref listing 0)
cleaned-price
cleaned-location
(aref url 0))))
(incf row-id))))
(format t "[[file:./working-data/2024-02-24-ideal-flatmate-manc.csv]]"))/craig.oates/overhaul2024/src/branch/master/working-data/2024-02-24-ideal-flatmate-manc.csv
The amount of listings on Ideal Flatmate is tiny compared to the others. I can basically show all them here.
csvlook working-data/2024-02-24-ideal-flatmate-manc.csv
# The output is in markdown format and not org-mode, so you will need to
# manually change the section divider, '| --- | --- |' to '|---+---|'.| ROW-ID | RAW-LISTING-INFO | RENT | LOCATION | URL |
|---|---|---|---|---|
| 0 | £690/month per roomChapel Street, Salford M3 5DZ, UK | 690 | Chapel Street Salford M3 5DZ UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113377 |
| 1 | £740/month per roomChapel Street, Salford M3 5DZ, UK | 740 | Chapel Street Salford M3 5DZ UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113378 |
| 2 | £841 - £842/month per roomMiddlewood Street, Salford, M5 4YW, UK | 841 | Middlewood Street Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120130 |
| 3 | £746 - £750/month per roomSalford, M5 4ZF, UK | 746 | Salford M5 4ZF UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id122936 |
| 4 | £200/month 100, 100 Lloyd Mansions, Salford M6 6HA, UK | 200 | NIL | https://www.idealflatmate.co.uk/spare-room/salford/property-id122970 |
| 5 | £488/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | 488 | Joshua Grange Pluto Cl Salford M6 6HF UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id122929 |
| 6 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | 580 | Joshua Grange Pluto Cl Salford M6 6HF UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id123025 |
| 7 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK | 480 | Greater Manchester Manchester M31 4HZ 296 UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id122962 |
| 8 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | 580 | Joshua Grange Pluto Cl Salford M6 6HF UK | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=242365&pid=123025 |
| 9 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK | 480 | Greater Manchester Manchester M31 4HZ 296 UK | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=210168&pid=122962 |
| 10 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK | 850 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121033 |
| 11 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK | 850 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121032 |
| 12 | £956 - £957/month per room7 Symphony Park, Manchester M1 7GB, UK | 956 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121034 |
| 13 | £980/month per room7 Symphony Park, Manchester M1 7GB, UK | 980 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121030 |
| 14 | £678 - £679/month per roomSalford M5 4YW, UK | 678 | Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120131 |
| 15 | £708 - £709/month per roomSalford M5 4YW, UK | 708 | Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120127 |
| 16 | £725/month per roomSalford M5 4YW, UK | 725 | Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120128 |
| 17 | £775/month per roomMiddlewood Street, Salford, M5 4YW, UK | 775 | Middlewood Street Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120129 |
- Min. Rent: £480
- Max. Rent: £956
As you can see, all the entries are listed /month. So, I've not bothered to
create a frequency column. I'm just going to move forward with that as the
default.
There are a could of entries in the URL column which have valid URL's but
point to the incorrect page on the website. I'm just going to leave them in and
manually go to their listings page on the website. There isn't enough incorrect
URL's to fix this in code. On top of that, I'm more interested in the average
rent price per month than the actual locations right now.
Explore CSV Data for Ideal Flatmate (2024-02-24)
The listing, in the table listed above, with the ROW-ID of 4, has rent
advertised at £200/month. This looked quite a bit off to the other
listings. So, I did a quick check and this is a student-only listing. Thus,
I'm going to remove it from the data, in an attempt to stop the average/mean
from being skewed.
; This is without the student-only listing removed.
(lisp-stat:defdf *im-manc*
(lisp-stat:read-csv #P"working-data/2024-02-24-ideal-flatmate-manc.csv"))#<DATA-FRAME:DATA-FRAME (18 observations of 5 variables)>
;; Removed the student listing.
(lisp-stat:defdf *im-manc-filt*
(lisp-stat:filter-rows *im-manc* '(> rent 201)))#<DATA-FRAME:DATA-FRAME (17 observations of 5 variables)>
(vega:defplot monthly-rents
`(:title "Rent Rates for Manchester on Ideal Flatmate (24/02/2024)"
:width 600
:height 600
:data ,*im-manc-filt*
:layer #((:mark (:type :bar)
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0))
:y (:field :rent :title "Rent (£)" :type :quantitative)
:tooltip (:field :rent)))
(:mark (:type rule :color "darkorange" :size 3)
:encoding (:y (:field :rent :type :quantitative :aggregate :average)
:tooltip (:field :rent :type :quantitative :aggregate :average))))))
(vega:write-html monthly-rents "renders/2024-02-24-ideal-flatmate-rent-manc.html")/craig.oates/overhaul2024/src/branch/master/renders/2024-02-24-ideal-flatmate-rent-manc.html
(format t "Mean Rent: £~a" (float (lisp-stat:mean *im-manc-filt*:rent)))Mean Rent: £714.5294
Summary of Ideal Flatmate Data
714.52 * 128574.24
Based on the average rent (inc. bill) for Ideal Flatmate, I would need to make about £9000/yr. This is just to cover living expenses. Up to now, I've been adding £5000 on top of that to cover travel, food, clothing and socialising costs.
9000 + 500014000
5000 / 12416.666666667
Based on the tax rates (see UK Wage and Tax Rates), the Personal Allowance is £12,570.
(let* ((earning-target 14000)
(p-allow 12570)
(taxable-income (- earning-target p-allow))
(tax-to-pay (* taxable-income 0.2))
(total (- earning-target tax-to-pay)))
(format t "- Annual Target Salary: £~a~%" earning-target)
(format t "- Part of Salary which is Taxable: £~a~%" taxable-income)
(format t "- Tax to Pay: £~a~%" tax-to-pay)
(format t "- Salary After Tax: £~a~%" total))- Annual Target Salary: £14000
- Part of Salary which is Taxable: £1430
- Tax to Pay: £286.0
- Salary After Tax: £13714.0
| Time Span | Value After Tax (£) | Mean Rent (£) |
|---|---|---|
| Annually | 13714 | 714.52 |
| Monthly (Before Rent) | 1142.8333 | |
| Monthly (After Rent) | 428.3133 | |
| Weekly (After Rent) | 107.07833 | |
| Daily (After Rent) | 15.296904 |
The spending limit of £15.29/day is, pretty much, on par with the assessment of the Spare Room listings: not great. And, at the time of writing, I've not taken into account the Income Tax with the Spare Room figures. Granted the difference won't be that different because the annual earnings are too low, for the most part (falling out of the Basic Rate band).