You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
300 lines
12 KiB
300 lines
12 KiB
3 months ago
|
#+options: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t
|
||
|
#+options: broken-links:nil c:nil creator:nil d:(not "LOGBOOK") date:t e:t
|
||
|
#+options: email:nil expand-links:t f:t inline:t num:t p:nil pri:nil prop:nil
|
||
|
#+options: stat:t tags:t tasks:t tex:t timestamp:t title:t toc:t todo:t |:t
|
||
|
#+title: Open Rent Manchester
|
||
|
#+date: \today
|
||
|
#+author: Craig Oates
|
||
|
#+email: craig@craigoates.net
|
||
|
#+language: en
|
||
|
#+select_tags: export
|
||
|
#+exclude_tags: noexport
|
||
|
#+creator: Emacs 29.1.90 (Org mode 9.7-pre)
|
||
|
#+cite_export:
|
||
|
|
||
|
* Setup Common Lisp Environment
|
||
|
|
||
|
I’ve copied the following code block over from other files. Run it if this is
|
||
|
your first file open in the session.
|
||
|
|
||
|
#+begin_src lisp :session :results silent
|
||
|
(ql:quickload :com.inuoe.jzon) ; JSON parser.
|
||
|
(ql:quickload :dexador) ; HTTP requests.
|
||
|
(ql:quickload :plump) ; HTML/XML parser.
|
||
|
(ql:quickload :lquery) ; HTML/DOM manipulation.
|
||
|
(ql:quickload :lparallel) ; Parallel programming.
|
||
|
(ql:quickload :cl-ppcre) ; RegEx. library.
|
||
|
(ql:quickload :plot/vega) ; Vega plotting library.
|
||
|
(ql:quickload :lisp-stat) ; Stat's library.
|
||
|
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
|
||
|
(ql:quickload :str) ; String library, expands on 'string' library.
|
||
|
#+end_src
|
||
|
|
||
|
* Gather Open Rent Data (Manually)
|
||
|
|
||
|
- [[https://www.openrent.co.uk][Open Rent]]
|
||
|
|
||
|
The site has an infinite-scroller feature, which means I can’t use ~curl~ to
|
||
|
grab a load of pages. With that said, the list isn’t as big as say
|
||
|
[[file:./spare-room-manchester.org][Spare Room (Manc.)]], so I can just save the page from within the browser and
|
||
|
parse that instead.
|
||
|
|
||
|
The filters I used:
|
||
|
|
||
|
- Date: 2024-03-02
|
||
|
- Location: Manchester
|
||
|
- Price Range: £400–£1500 pcm
|
||
|
- Proximity: 15 km
|
||
|
- No student listings
|
||
|
- Only monthly rents listed (pcm)
|
||
|
|
||
|
The website says it’s showing 120 of 741 properties found – with the filters
|
||
|
applied.
|
||
|
|
||
|
The data is saved in the =raw-data/external/2024-03-02-open-rent-manc/=
|
||
|
directory. This data will not be committed to the repositories commit history,
|
||
|
as is the standard I set for other websites.
|
||
|
|
||
|
* Clean Up and Parse Data
|
||
|
|
||
|
Going to separate out the listing into their own HTML files and then build a CSV
|
||
|
file of that data. This is standard practice I've established in other
|
||
|
(Manchester based) data exploration files.
|
||
|
|
||
|
#+begin_src shell :results silent
|
||
|
mkdir raw-data/external/2024-03-02-open-rent-manc-listings/
|
||
|
#+end_src
|
||
|
|
||
|
#+begin_src lisp :results silent
|
||
|
(let ((counter 0))
|
||
|
(loop for file-path
|
||
|
in (directory #P"raw-data/external/2024-03-02-open-rent-manc/*.html")
|
||
|
do (with-open-file (in-stream file-path)
|
||
|
(let* ((doc (plump:parse in-stream))
|
||
|
;; .listing-info
|
||
|
(listings (lquery:$ doc "a.pli.clearfix" (serialize))))
|
||
|
(loop for item across listings
|
||
|
do (let ((out-path
|
||
|
(merge-pathnames
|
||
|
#P"raw-data/external/2024-03-02-open-rent-manc-listings/"
|
||
|
(format nil "listing-~a.html" (write-to-string counter)))))
|
||
|
(with-open-file (out-stream
|
||
|
out-path
|
||
|
:direction :output
|
||
|
:if-exists :supersede)
|
||
|
(format out-stream "~a~%" item))
|
||
|
(incf counter)))))))
|
||
|
#+end_src
|
||
|
|
||
|
The site has a ‘Let Agreed’ section, which is of no use to me here. They are
|
||
|
listing which have already been taken. So, I will need to remove them from the
|
||
|
listings.
|
||
|
|
||
|
#+begin_src shell :results output
|
||
|
#!/bin/bash
|
||
|
|
||
|
target_string="Let Agreed"
|
||
|
directory="raw-data/external/2024-03-02-open-rent-manc-listings"
|
||
|
|
||
|
# Loop over all files in the directory
|
||
|
for file in "$directory"/*
|
||
|
do
|
||
|
# If the target string is found in the file, delete the file
|
||
|
if grep -q "$target_string" "$file"
|
||
|
then
|
||
|
rm "$file"
|
||
|
echo "Deleted file: $file"
|
||
|
fi
|
||
|
done
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
: Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-114.html
|
||
|
: Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-115.html
|
||
|
: Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-116.html
|
||
|
: Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-117.html
|
||
|
: Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-118.html
|
||
|
: Deleted file: raw-data/external/2024-03-02-open-rent-manc-listings/listing-119.html
|
||
|
|
||
|
Deleting those files leaves,
|
||
|
|
||
|
#+begin_src shell :results output
|
||
|
COUNT=$(ls -1 "raw-data/external/2024-03-02-open-rent-manc-listings/" | wc -l)
|
||
|
echo "There are $COUNT files left in the directory."
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
: There are 114 files left in the directory.
|
||
|
|
||
|
|
||
|
The 114 files left and the 6 files deleted bring the total amount of listings up
|
||
|
to 120, which is what the website stated is was displaying. So, all looks good
|
||
|
and ready to move forward.
|
||
|
|
||
|
** Create CSV of Listings
|
||
|
|
||
|
#+begin_src lisp :results output raw
|
||
|
(let ((row-id 0)
|
||
|
(out-file #P"working-data/2024-03-02-open-rent-manc.csv"))
|
||
|
(with-open-file (out-stream
|
||
|
out-file
|
||
|
:direction :output
|
||
|
:if-exists :supersede)
|
||
|
(format out-stream "ROW-ID,PRICE,LOCATION,URL~%")
|
||
|
(loop for input
|
||
|
in (directory #P"raw-data/external/2024-03-02-open-rent-manc-listings/*.html")
|
||
|
do (with-open-file (in-stream input)
|
||
|
(let* ((doc (plump:parse in-stream))
|
||
|
(price (lquery:$ doc ".pl-title" (text)))
|
||
|
;; e.g. Transform '£1,250 per month' to '1250'.
|
||
|
(cleaned-price
|
||
|
(first
|
||
|
(cl-ppcre:all-matches-as-strings
|
||
|
"\\d+"
|
||
|
(cl-ppcre:regex-replace-all
|
||
|
"[\\s\\n]+"
|
||
|
(format nil "~s" (str:replace-all "," "" (aref price 0)))
|
||
|
" "))))
|
||
|
(location (lquery:$ doc ".listing-title" (text)))
|
||
|
(link (lquery:$ doc "a" (attr :href))))
|
||
|
(format out-stream "~d,~d,~a,~a~%"
|
||
|
row-id
|
||
|
cleaned-price
|
||
|
(str:replace-all "," " " (aref location 0))
|
||
|
(aref link 0))))
|
||
|
(incf row-id)))
|
||
|
(format t "[[file:./~a]]" out-file))
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
[[file:./working-data/2024-03-02-open-rent-manc.csv]]
|
||
|
|
||
|
There is nothing which stands out in the CSV file which looks like it needs
|
||
|
looking at manually. Going to start exploring the data.
|
||
|
|
||
|
* Explore CSV Data for Open Rent (2024-03-02
|
||
|
|
||
|
#+begin_src lisp :session
|
||
|
(lisp-stat:defdf *or-manc*
|
||
|
(lisp-stat:read-csv #P"working-data/2024-03-02-open-rent-manc.csv"))
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
: #<DATA-FRAME:DATA-FRAME (114 observations of 4 variables)>
|
||
|
|
||
|
The number entries listed in the file is a bit to big for here, so going to just
|
||
|
show the head of the file.
|
||
|
|
||
|
#+begin_src shell :results output raw
|
||
|
head -n 11 working-data/2024-03-02-open-rent-manc.csv | csvlook
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
| ROW-ID | PRICE | LOCATION | URL |
|
||
|
|--------+-------+------------------------------------------------------------------+------------------------------------|
|
||
|
| 0 | 400 | Room in a Shared House Manor Road M32 | https://www.openrent.co.uk/1862286 |
|
||
|
| 1 | 450 | Room in a Shared Flat Near Uni Oxford Road Piccadilly Stat M12 | https://www.openrent.co.uk/1999077 |
|
||
|
| 2 | 500 | Room in a Shared House Whiteway Street M9 | https://www.openrent.co.uk/2001232 |
|
||
|
| 3 | 1,000 | Room in a Shared House Ashfield Road M13 | https://www.openrent.co.uk/1976038 |
|
||
|
| 4 | 1,050 | Room in a Shared House Adelphi Wharf 1 M3 | https://www.openrent.co.uk/1916001 |
|
||
|
| 5 | 1,100 | 1 Bed Flat Teal Close WA14 | https://www.openrent.co.uk/2000362 |
|
||
|
| 6 | 1,150 | 1 Bed Flat Green Quarter M3 | https://www.openrent.co.uk/256714 |
|
||
|
| 7 | 1,150 | Room in a Shared Flat Water Street M3 | https://www.openrent.co.uk/1997609 |
|
||
|
| 8 | 1,200 | Room in a Shared House Noor Gardens M16 | https://www.openrent.co.uk/1967416 |
|
||
|
| 9 | 1,200 | 2 Bed Flat Swinton M28 | https://www.openrent.co.uk/1968257 |
|
||
|
|
||
|
#+begin_src lisp :session :results file
|
||
|
(vega:defplot or-montly-manc
|
||
|
`(:title "Rent Rates (Bills Inc.) for Manchester on Open Rent (02/03/2024)"
|
||
|
:description "A breakdown of the rent rates for listings on Open Rent, in Manchester (02/03/2024)"
|
||
|
:data ,*or-manc*
|
||
|
:width 1900
|
||
|
:height 1000
|
||
|
:layer #((:mark (:type :bar)
|
||
|
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal)
|
||
|
:y (:field :price :title "Monthly Rent (£)" :type :quantitative)
|
||
|
:tooltip (:field :price)))
|
||
|
(:mark (:type :rule :color "darkorange" :size 3)
|
||
|
:encoding (:y (:field :price :type :quantitative :aggregate :average)
|
||
|
:tooltip (:field :price :type :quantitative :aggregate :average))))))
|
||
|
(vega:write-html or-montly-manc "renders/2024-03-02-open-rent-manc.html")
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
[[file:renders/2024-03-02-open-rent-manc.html]]
|
||
|
|
||
|
#+begin_src shell :results silent
|
||
|
mv ~/Downloads/visualization.png ./renders/2024-03-02-open-rent-manc.png
|
||
|
#+end_src
|
||
|
|
||
|
[[file:./renders/2024-03-02-open-rent-manc.png]]
|
||
|
|
||
|
#+begin_src lisp :session :results output code raw
|
||
|
(format t "- Mean Rent: £ ~a~%" (float (lisp-stat:mean *or-manc*:price)))
|
||
|
(format t "- Min. Rent: £ ~d~%" (reduce #'min *or-manc*:price))
|
||
|
(format t "- Max. Rent: £ ~d" (reduce #'max *or-manc*:price))
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
- Mean Rent: £ 735.4123
|
||
|
- Min. Rent: £ 400
|
||
|
- Max. Rent: £ 1350
|
||
|
|
||
|
#+begin_src calc :results ouput
|
||
|
735.41 * 12
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
: 8824.92
|
||
|
|
||
|
* Summary of Open Rent
|
||
|
|
||
|
Based on the average rent for Open Rent, I would need to make around £9,000/yr
|
||
|
to cover my living costs. This does not include travel, food, clothing,
|
||
|
socialising or Income Tax payments.
|
||
|
|
||
|
Most listings, if not all, are advertised as a house share. This puts Open Rent
|
||
|
more inline with [[file:./spare-room-manchester.org][Spare Room]] than the others.
|
||
|
|
||
|
At this point, I've established a practice of add £5,000 to the annual average
|
||
|
rent total. This is to get a baseline idea of how bare-bones my income needs to
|
||
|
be, for me to survive.
|
||
|
|
||
|
#+begin_src calc :results output
|
||
|
8824.92 + 5000
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
: 13824.92
|
||
|
|
||
|
Using £13824.92 as the starting point (see [[file:./uk-wage-tax.org][UK Wage and Tax Rated]],
|
||
|
|
||
|
#+begin_src lisp :results output raw
|
||
|
(let* ((earning-target 13824.94)
|
||
|
(p-allow 12570)
|
||
|
(taxable-income (- earning-target p-allow))
|
||
|
(tax-to-pay (* taxable-income 0.2))
|
||
|
(total (- earning-target tax-to-pay)))
|
||
|
(format t "- Annual Target Salary: £~a~%" earning-target)
|
||
|
(format t "- Part of Salary which is Taxable: £~a~%" taxable-income)
|
||
|
(format t "- Tax to Pay: £~a~%" tax-to-pay)
|
||
|
(format t "- Salary After Tax: £~a~%" total))
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
- Annual Target Salary: £13824.94
|
||
|
- Part of Salary which is Taxable: £1254.9404
|
||
|
- Tax to Pay: £250.98808
|
||
|
- Salary After Tax: £13573.952
|
||
|
|
||
|
| Time Span | Value After Tax (£) | Mean Rent (£) |
|
||
|
|-----------------------+---------------------+---------------|
|
||
|
| Annually | 13573.95 | 735.41 |
|
||
|
| Monthly (Before Rent) | 1131.1625 | |
|
||
|
| Monthly (After Rent) | 395.7525 | |
|
||
|
| Weekly (After Rent) | 98.938125 | |
|
||
|
| Daily (After Rent) | 14.134018 | |
|
||
|
#+TBLFM: @3$2=@-1/12::@4$2=@-1-@-2$+1::@5$2=@-1/4::@6$2=@-1/7
|
||
|
|
||
|
So, a spending limit of £14.13/day.
|