You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
249 lines
10 KiB
249 lines
10 KiB
3 months ago
|
#+options: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t
|
||
|
#+options: broken-links:nil c:nil creator:nil d:(not "LOGBOOK") date:t e:t
|
||
|
#+options: email:nil expand-links:t f:t inline:t num:t p:nil pri:nil prop:nil
|
||
|
#+options: stat:t tags:t tasks:t tex:t timestamp:t title:t toc:t todo:t |:t
|
||
|
#+title: On The Market Manchester
|
||
|
#+date: \today
|
||
|
#+author: Craig Oates
|
||
|
#+email: craig@craigoates.net
|
||
|
#+language: en
|
||
|
#+select_tags: export
|
||
|
#+exclude_tags: noexport
|
||
|
#+creator: Emacs 29.1.90 (Org mode 9.7-pre)
|
||
|
#+cite_export:
|
||
|
|
||
|
* Setup Common Lisp Environment
|
||
|
|
||
|
I’ve copied the following code block over from other files. Run it if this is
|
||
|
your first file open in the session.
|
||
|
|
||
|
#+begin_src lisp :session :results silent
|
||
|
(ql:quickload :com.inuoe.jzon) ; JSON parser.
|
||
|
(ql:quickload :dexador) ; HTTP requests.
|
||
|
(ql:quickload :plump) ; HTML/XML parser.
|
||
|
(ql:quickload :lquery) ; HTML/DOM manipulation.
|
||
|
(ql:quickload :lparallel) ; Parallel programming.
|
||
|
(ql:quickload :cl-ppcre) ; RegEx. library.
|
||
|
(ql:quickload :plot/vega) ; Vega plotting library.
|
||
|
(ql:quickload :lisp-stat) ; Stat's library.
|
||
|
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
|
||
|
(ql:quickload :str) ; String library, expands on 'string' library.
|
||
|
#+end_src
|
||
|
|
||
|
* Gather On The Market Data
|
||
|
|
||
|
- [[https:/www.onthemarket.com][On The Market]]
|
||
|
|
||
|
Having had a quick look on the website, it says there are 438 results matching
|
||
|
the following criteria,
|
||
|
|
||
|
- Location: Manchester, Greater Manchester
|
||
|
- To rent
|
||
|
- Max: £1,2500 pcm
|
||
|
- Min: £300 pcm (to filter out car parks)
|
||
|
- No student accommodation
|
||
|
|
||
|
There are too many results to grab the data manually, so I'm gonna have to
|
||
|
scrape it with some code. The website is paginated and the last page for these
|
||
|
results is ~page=18~, as stated in the URL (19 pages but starts from 0).
|
||
|
|
||
|
#+begin_src shell :results silent
|
||
|
cd raw-data/external
|
||
|
DIRECTORY="$(date '+%Y-%m-%d')-on-the-market-manc"
|
||
|
mkdir $DIRECTORY
|
||
|
for PAGE in {0..18}
|
||
|
do
|
||
|
curl -o "$DIRECTORY/on-the-market-$PAGE.html" \
|
||
|
"https://www.onthemarket.com/to-rent/property/manchester/?direction=asc&max-price=1250&min-price=300&page=$PAGE&sort-field=price&student=false"
|
||
|
sleep 5
|
||
|
done
|
||
|
# Change back to the project's root directory, so I don't call code whilst still
|
||
|
# in this directory.
|
||
|
cd ../../
|
||
|
#+end_src
|
||
|
|
||
|
* Clean Up and Parse Data
|
||
|
|
||
|
#+begin_src shell :results silent
|
||
|
mkdir raw-data/external/2024-03-01-on-the-market-manc-listings/
|
||
|
#+end_src
|
||
|
|
||
|
#+begin_src lisp :results silent
|
||
|
(let ((counter 0))
|
||
|
(loop for file-path
|
||
|
in (directory #P"raw-data/external/2024-03-01-on-the-market-manc/*.html")
|
||
|
do (with-open-file (in-stream file-path)
|
||
|
(let* ((doc (plump:parse in-stream))
|
||
|
(listings (lquery:$ doc ".otm-PropertyCardInfo" (serialize))))
|
||
|
(loop for item across listings
|
||
|
do (let ((out-path
|
||
|
(merge-pathnames "raw-data/external/2024-03-01-on-the-market-manc-listings/"
|
||
|
(format nil "listing-~a.html" (write-to-string counter)))))
|
||
|
(with-open-file (out-stream
|
||
|
out-path
|
||
|
:direction :output
|
||
|
:if-exists :supersede)
|
||
|
(format out-stream "~a" item))
|
||
|
(incf counter)))))))
|
||
|
#+end_src
|
||
|
|
||
|
* Create CSV of Listings
|
||
|
|
||
|
#+begin_src lisp :results output
|
||
|
(with-open-file (out-stream
|
||
|
#P"working-data/2024-03-01-on-the-market-manc.csv"
|
||
|
:direction :output
|
||
|
:if-exists :supersede)
|
||
|
(let ((row-id 0))
|
||
|
(format out-stream "ROW-ID,LOCATION,RENT,URL,RAW-PRICE,RAW-LISTING~%")
|
||
|
(loop for filepath
|
||
|
in (directory #P"raw-data/external/2024-03-01-on-the-market-manc-listings/*.html")
|
||
|
do (with-open-file (in-stream filepath)
|
||
|
(let* ((doc (plump:parse in-stream))
|
||
|
(raw-price (lquery:$ doc ".otm-Price" (text)))
|
||
|
(cleaned-price (first (cl-ppcre:all-matches-as-strings "[0-9,]+" (aref raw-price 0))))
|
||
|
(raw-listing (lquery:$ doc "p" (text)))
|
||
|
(link (lquery:$ doc "a" (attr :href)))
|
||
|
(address (lquery:$ doc ".address" (text))))
|
||
|
(format t "~a~%" cleaned-price)
|
||
|
(format out-stream "~d,~a,~d,~a,~a,~a~%"
|
||
|
row-id
|
||
|
(str:replace-all "," " " (aref address 0))
|
||
|
(str:replace-all "," "" cleaned-price)
|
||
|
(format nil "https://www.onthemarket.com~a" (aref link 0))
|
||
|
(str:replace-all "," "" (aref raw-price 0))
|
||
|
(str:replace-all "," " " (aref raw-listing 0)))))
|
||
|
(incf row-id))))
|
||
|
#+end_src
|
||
|
|
||
|
I've gone over the CSV file and all the entries are listed with monthly rent
|
||
|
rates (and a weekly breakdown next to them). Because of this, I don't need to
|
||
|
create weekly and monthly breakdowns of the listing, to find the average
|
||
|
advertised rent.
|
||
|
|
||
|
I manually removed a listing for a garage (listed over £400 pcm). It was only
|
||
|
one and it was easier to do than code something up. A quick find-and-replace
|
||
|
search show no entries for 'car park' or 'garage' after that.
|
||
|
|
||
|
The amount of listings in the file is too large to list here. So, here's the
|
||
|
link to the CSV file,
|
||
|
|
||
|
- [[file:./working-data/2024-03-01-on-the-market-manc.csv]]
|
||
|
|
||
|
* Explore CSV Data for On The Market (2024-03-01)
|
||
|
|
||
|
#+begin_src lisp :session
|
||
|
(lisp-stat:defdf *otm-manc*
|
||
|
(lisp-stat:read-csv #P"working-data/2024-03-01-on-the-market-manc.csv"))
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
: #<DATA-FRAME:DATA-FRAME (438 observations of 6 variables)>
|
||
|
|
||
|
The data is in pretty good shape, so don't need to filter out any entries in the
|
||
|
data-frame.
|
||
|
|
||
|
#+begin_src lisp :session :results file
|
||
|
(vega:defplot otm-monthly-manc
|
||
|
`(:title "Rent Rates for Manchester for On The Market (01/03/2024)"
|
||
|
:description "A breakdown of the rent rates, for On The Markert, in Manchester (01/03/2024)."
|
||
|
:data ,*otm-manc*
|
||
|
:width 1200
|
||
|
:height 1000
|
||
|
:layer #((:mark (:type :bar)
|
||
|
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0))
|
||
|
:y (:field :rent :title "Monthly Rent with Bills Inc. (£)" :type :quantitative)
|
||
|
:tooltip (:field :rent)))
|
||
|
(:mark (:type :rule :color "darkorange" :size 3)
|
||
|
:encoding (:y (:field :rent :type :quantitative :aggregate :average)
|
||
|
:tooltip (:field :rent :type :quantitative :aggregate :average))))))
|
||
|
(vega:write-html otm-monthly-manc "renders/2024-03-01-on-the-market-rent-manc.html")
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
[[file:renders/2024-03-01-on-the-market-rent-manc.html]]
|
||
|
|
||
|
#+begin_src shell
|
||
|
mv ~/Downloads/visualization.png ./renders/2024-03-01-on-the-market-rent-manc.png
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
|
||
|
[[file:./renders/2024-03-01-on-the-market-rent-manc.png]]
|
||
|
|
||
|
#+begin_src lisp :session :results output code raw
|
||
|
(format t "- Mean Rent: £ ~a~%" (float (lisp-stat:mean *otm-manc*:rent)))
|
||
|
(format t "- Min. Rent: £ ~d~%" (reduce #'min *otm-manc*:rent))
|
||
|
(format t "- Max. Rent: £ ~d" (reduce #'max *otm-manc*:rent))
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
- Mean Rent: £ 1005.5388
|
||
|
- Min. Rent: £ 395
|
||
|
- Max. Rent: £ 1250
|
||
|
|
||
|
* Summary of On The Market
|
||
|
|
||
|
#+begin_src calc :results output
|
||
|
1005.53 * 12
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
: 12066.36
|
||
|
|
||
|
Based on the average rent for On The Market, I would need to make around
|
||
|
£13,000/yr to cover my living costs. This does not include travel, food,
|
||
|
clothing, socialising or Income Tax payments.
|
||
|
|
||
|
*NOTE: ON THE MARKET DOESN'T MAKE IT CLEAR IF BILLS ARE INCLUDED*. On top of that,
|
||
|
I noticed some listing stated they are student accommodation only, even though I
|
||
|
used the filter option to remove them. This is more the agents abusing the
|
||
|
system, though. It's not apparent until you start reading the descriptions on
|
||
|
the listing specific pages. Unfortunately, I can't filter these listings out in
|
||
|
a practical way, via code. And, there are too many listings to go through
|
||
|
one-by-one and remove them from the CSV file.
|
||
|
|
||
|
As standard now, I'll add £5,000 to the basic annual rent calculated above.
|
||
|
|
||
|
#+begin_src calc :results output
|
||
|
12066.36 + 5000
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
: 17066.36
|
||
|
|
||
|
Using £17066.36 as the starting point (see [[file:./uk-wage-tax.org][UK Wage and Tax Rates]]),
|
||
|
|
||
|
#+begin_src lisp :results output raw
|
||
|
(let* ((earning-target 17066.36)
|
||
|
(p-allow 12570)
|
||
|
(taxable-income (- earning-target p-allow))
|
||
|
(tax-to-pay (* taxable-income 0.2))
|
||
|
(total (- earning-target tax-to-pay)))
|
||
|
(format t "- Annual Target Salary: £~a~%" earning-target)
|
||
|
(format t "- Part of Salary which is Taxable: £~a~%" taxable-income)
|
||
|
(format t "- Tax to Pay: £~a~%" tax-to-pay)
|
||
|
(format t "- Salary After Tax: £~a~%" total))
|
||
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
- Annual Target Salary: £17066.36
|
||
|
- Part of Salary which is Taxable: £4496.3594
|
||
|
- Tax to Pay: £899.2719
|
||
|
- Salary After Tax: £16167.088
|
||
|
|
||
|
| Time Span | Value After Tax (£) | Mean Rent (£) |
|
||
|
|-----------------------+---------------------+---------------|
|
||
|
| Annually | 16167.09 | 1005.53 |
|
||
|
| Monthly (Before Rent) | 1347.2575 | |
|
||
|
| Monthly (After Rent) | 341.7275 | |
|
||
|
| Weekly (After Rent) | 85.431875 | |
|
||
|
| Daily (After Rent) | 12.204554 | |
|
||
|
#+TBLFM: @3$2=@-1/12::@4$2=@-1-@-2$+1::@5$2=@-1/4::@6$2=@-1/7
|
||
|
|
||
|
It looks like using On The Market is the worse route to go down, at the time of
|
||
|
writing. Having just £12.20/day to spend after paying my rent is the lowest so
|
||
|
far (2024-03-01 Fri). On top of that On The Market listings not saying if the
|
||
|
bills are included, they don't make is clear and obvious at least. So, there is
|
||
|
a strong chance this estimate is way off and the bills will be too much.
|