12 KiB
Open Rent Manchester
- Setup Common Lisp Environment
- Gather Open Rent Data (Manually)
- Clean Up and Parse Data
- Explore CSV Data for Open Rent (2024-03-02
- Summary of Open Rent
Setup Common Lisp Environment
I’ve copied the following code block over from other files. Run it if this is your first file open in the session.
(ql:quickload :com.inuoe.jzon) ; JSON parser.
(ql:quickload :dexador) ; HTTP requests.
(ql:quickload :plump) ; HTML/XML parser.
(ql:quickload :lquery) ; HTML/DOM manipulation.
(ql:quickload :lparallel) ; Parallel programming.
(ql:quickload :cl-ppcre) ; RegEx. library.
(ql:quickload :plot/vega) ; Vega plotting library.
(ql:quickload :lisp-stat) ; Stat's library.
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
(ql:quickload :str) ; String library, expands on 'string' library.Gather Open Rent Data (Manually)
The site has an infinite-scroller feature, which means I can’t use curl to
grab a load of pages. With that said, the list isn’t as big as say
Spare Room (Manc.), so I can just save the page from within the browser and
parse that instead.
The filters I used:
- Date: 2024-03-02
- Location: Manchester
- Price Range: £400–£1500 pcm
- Proximity: 15 km
- No student listings
- Only monthly rents listed (pcm)
The website says it’s showing 120 of 741 properties found – with the filters applied.
The data is saved in the raw-data/external/2024-03-02-open-rent-manc/
directory. This data will not be committed to the repositories commit history,
as is the standard I set for other websites.
Clean Up and Parse Data
Going to separate out the listing into their own HTML files and then build a CSV file of that data. This is standard practice I've established in other (Manchester based) data exploration files.
mkdir raw-data/external/2024-03-02-open-rent-manc-listings/ (let ((counter 0))
(loop for file-path
in (directory #P"raw-data/external/2024-03-02-open-rent-manc/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
;; .listing-info
(listings (lquery:$ doc "a.pli.clearfix" (serialize))))
(loop for item across listings
do (let ((out-path
(merge-pathnames
#P"raw-data/external/2024-03-02-open-rent-manc-listings/"
(format nil "listing-~a.html" (write-to-string counter)))))
(with-open-file (out-stream
out-path
:direction :output
:if-exists :supersede)
(format out-stream "~a~%" item))
(incf counter)))))))The site has a ‘Let Agreed’ section, which is of no use to me here. They are listing which have already been taken. So, I will need to remove them from the listings.
#!/bin/bash
target_string="Let Agreed"
directory="raw-data/external/2024-03-02-open-rent-manc-listings"
# Loop over all files in the directory
for file in "$directory"/*
do
# If the target string is found in the file, delete the file
if grep -q "$target_string" "$file"
then
rm "$file"
echo "Deleted file: $file"
fi
doneDeleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-114.html Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-115.html Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-116.html Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-117.html Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-118.html Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-119.html
Deleting those files leaves,
COUNT=$(ls -1 "raw-data/external/2024-03-02-open-rent-manc-listings/" | wc -l)
echo "There are $COUNT files left in the directory."There are 114 files left in the directory.
The 114 files left and the 6 files deleted bring the total amount of listings up to 120, which is what the website stated is was displaying. So, all looks good and ready to move forward.
Create CSV of Listings
(let ((row-id 0)
(out-file #P"working-data/2024-03-02-open-rent-manc.csv"))
(with-open-file (out-stream
out-file
:direction :output
:if-exists :supersede)
(format out-stream "ROW-ID,PRICE,LOCATION,URL~%")
(loop for input
in (directory #P"raw-data/external/2024-03-02-open-rent-manc-listings/*.html")
do (with-open-file (in-stream input)
(let* ((doc (plump:parse in-stream))
(price (lquery:$ doc ".pl-title" (text)))
;; e.g. Transform '£1,250 per month' to '1250'.
(cleaned-price
(first
(cl-ppcre:all-matches-as-strings
"\\d+"
(cl-ppcre:regex-replace-all
"[\\s\\n]+"
(format nil "~s" (str:replace-all "," "" (aref price 0)))
" "))))
(location (lquery:$ doc ".listing-title" (text)))
(link (lquery:$ doc "a" (attr :href))))
(format out-stream "~d,~d,~a,~a~%"
row-id
cleaned-price
(str:replace-all "," " " (aref location 0))
(aref link 0))))
(incf row-id)))
(format t "[[file:./~a]]" out-file))/craig.oates/overhaul2024/src/branch/master/working-data/2024-03-02-open-rent-manc.csv
There is nothing which stands out in the CSV file which looks like it needs looking at manually. Going to start exploring the data.
Explore CSV Data for Open Rent (2024-03-02
(lisp-stat:defdf *or-manc*
(lisp-stat:read-csv #P"working-data/2024-03-02-open-rent-manc.csv"))#<DATA-FRAME:DATA-FRAME (114 observations of 4 variables)>
The number entries listed in the file is a bit to big for here, so going to just show the head of the file.
head -n 11 working-data/2024-03-02-open-rent-manc.csv | csvlook| ROW-ID | PRICE | LOCATION | URL |
|---|---|---|---|
| 0 | 400 | Room in a Shared House Manor Road M32 | https://www.openrent.co.uk/1862286 |
| 1 | 450 | Room in a Shared Flat Near Uni Oxford Road Piccadilly Stat M12 | https://www.openrent.co.uk/1999077 |
| 2 | 500 | Room in a Shared House Whiteway Street M9 | https://www.openrent.co.uk/2001232 |
| 3 | 1,000 | Room in a Shared House Ashfield Road M13 | https://www.openrent.co.uk/1976038 |
| 4 | 1,050 | Room in a Shared House Adelphi Wharf 1 M3 | https://www.openrent.co.uk/1916001 |
| 5 | 1,100 | 1 Bed Flat Teal Close WA14 | https://www.openrent.co.uk/2000362 |
| 6 | 1,150 | 1 Bed Flat Green Quarter M3 | https://www.openrent.co.uk/256714 |
| 7 | 1,150 | Room in a Shared Flat Water Street M3 | https://www.openrent.co.uk/1997609 |
| 8 | 1,200 | Room in a Shared House Noor Gardens M16 | https://www.openrent.co.uk/1967416 |
| 9 | 1,200 | 2 Bed Flat Swinton M28 | https://www.openrent.co.uk/1968257 |
(vega:defplot or-montly-manc
`(:title "Rent Rates (Bills Inc.) for Manchester on Open Rent (02/03/2024)"
:description "A breakdown of the rent rates for listings on Open Rent, in Manchester (02/03/2024)"
:data ,*or-manc*
:width 1900
:height 1000
:layer #((:mark (:type :bar)
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal)
:y (:field :price :title "Monthly Rent (£)" :type :quantitative)
:tooltip (:field :price)))
(:mark (:type :rule :color "darkorange" :size 3)
:encoding (:y (:field :price :type :quantitative :aggregate :average)
:tooltip (:field :price :type :quantitative :aggregate :average))))))
(vega:write-html or-montly-manc "renders/2024-03-02-open-rent-manc.html")/craig.oates/overhaul2024/src/branch/master/renders/2024-03-02-open-rent-manc.html
mv ~/Downloads/visualization.png ./renders/2024-03-02-open-rent-manc.png
(format t "- Mean Rent: £ ~a~%" (float (lisp-stat:mean *or-manc*:price)))
(format t "- Min. Rent: £ ~d~%" (reduce #'min *or-manc*:price))
(format t "- Max. Rent: £ ~d" (reduce #'max *or-manc*:price))- Mean Rent: £ 735.4123
- Min. Rent: £ 400
- Max. Rent: £ 1350
735.41 * 128824.92
Summary of Open Rent
Based on the average rent for Open Rent, I would need to make around £9,000/yr to cover my living costs. This does not include travel, food, clothing, socialising or Income Tax payments.
Most listings, if not all, are advertised as a house share. This puts Open Rent more inline with Spare Room than the others.
At this point, I've established a practice of add £5,000 to the annual average rent total. This is to get a baseline idea of how bare-bones my income needs to be, for me to survive.
8824.92 + 500013824.92
Using £13824.92 as the starting point (see UK Wage and Tax Rated,
(let* ((earning-target 13824.94)
(p-allow 12570)
(taxable-income (- earning-target p-allow))
(tax-to-pay (* taxable-income 0.2))
(total (- earning-target tax-to-pay)))
(format t "- Annual Target Salary: £~a~%" earning-target)
(format t "- Part of Salary which is Taxable: £~a~%" taxable-income)
(format t "- Tax to Pay: £~a~%" tax-to-pay)
(format t "- Salary After Tax: £~a~%" total))- Annual Target Salary: £13824.94
- Part of Salary which is Taxable: £1254.9404
- Tax to Pay: £250.98808
- Salary After Tax: £13573.952
| Time Span | Value After Tax (£) | Mean Rent (£) |
|---|---|---|
| Annually | 13573.95 | 735.41 |
| Monthly (Before Rent) | 1131.1625 | |
| Monthly (After Rent) | 395.7525 | |
| Weekly (After Rent) | 98.938125 | |
| Daily (After Rent) | 14.134018 |
So, a spending limit of £14.13/day.