1 changed files with 248 additions and 0 deletions
@ -0,0 +1,248 @@
|
||||
#+options: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t |
||||
#+options: broken-links:nil c:nil creator:nil d:(not "LOGBOOK") date:t e:t |
||||
#+options: email:nil expand-links:t f:t inline:t num:t p:nil pri:nil prop:nil |
||||
#+options: stat:t tags:t tasks:t tex:t timestamp:t title:t toc:t todo:t |:t |
||||
#+title: On The Market Manchester |
||||
#+date: \today |
||||
#+author: Craig Oates |
||||
#+email: craig@craigoates.net |
||||
#+language: en |
||||
#+select_tags: export |
||||
#+exclude_tags: noexport |
||||
#+creator: Emacs 29.1.90 (Org mode 9.7-pre) |
||||
#+cite_export: |
||||
|
||||
* Setup Common Lisp Environment |
||||
|
||||
I’ve copied the following code block over from other files. Run it if this is |
||||
your first file open in the session. |
||||
|
||||
#+begin_src lisp :session :results silent |
||||
(ql:quickload :com.inuoe.jzon) ; JSON parser. |
||||
(ql:quickload :dexador) ; HTTP requests. |
||||
(ql:quickload :plump) ; HTML/XML parser. |
||||
(ql:quickload :lquery) ; HTML/DOM manipulation. |
||||
(ql:quickload :lparallel) ; Parallel programming. |
||||
(ql:quickload :cl-ppcre) ; RegEx. library. |
||||
(ql:quickload :plot/vega) ; Vega plotting library. |
||||
(ql:quickload :lisp-stat) ; Stat's library. |
||||
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy. |
||||
(ql:quickload :str) ; String library, expands on 'string' library. |
||||
#+end_src |
||||
|
||||
* Gather On The Market Data |
||||
|
||||
- [[https:/www.onthemarket.com][On The Market]] |
||||
|
||||
Having had a quick look on the website, it says there are 438 results matching |
||||
the following criteria, |
||||
|
||||
- Location: Manchester, Greater Manchester |
||||
- To rent |
||||
- Max: £1,2500 pcm |
||||
- Min: £300 pcm (to filter out car parks) |
||||
- No student accommodation |
||||
|
||||
There are too many results to grab the data manually, so I'm gonna have to |
||||
scrape it with some code. The website is paginated and the last page for these |
||||
results is ~page=18~, as stated in the URL (19 pages but starts from 0). |
||||
|
||||
#+begin_src shell :results silent |
||||
cd raw-data/external |
||||
DIRECTORY="$(date '+%Y-%m-%d')-on-the-market-manc" |
||||
mkdir $DIRECTORY |
||||
for PAGE in {0..18} |
||||
do |
||||
curl -o "$DIRECTORY/on-the-market-$PAGE.html" \ |
||||
"https://www.onthemarket.com/to-rent/property/manchester/?direction=asc&max-price=1250&min-price=300&page=$PAGE&sort-field=price&student=false" |
||||
sleep 5 |
||||
done |
||||
# Change back to the project's root directory, so I don't call code whilst still |
||||
# in this directory. |
||||
cd ../../ |
||||
#+end_src |
||||
|
||||
* Clean Up and Parse Data |
||||
|
||||
#+begin_src shell :results silent |
||||
mkdir raw-data/external/2024-03-01-on-the-market-manc-listings/ |
||||
#+end_src |
||||
|
||||
#+begin_src lisp :results silent |
||||
(let ((counter 0)) |
||||
(loop for file-path |
||||
in (directory #P"raw-data/external/2024-03-01-on-the-market-manc/*.html") |
||||
do (with-open-file (in-stream file-path) |
||||
(let* ((doc (plump:parse in-stream)) |
||||
(listings (lquery:$ doc ".otm-PropertyCardInfo" (serialize)))) |
||||
(loop for item across listings |
||||
do (let ((out-path |
||||
(merge-pathnames "raw-data/external/2024-03-01-on-the-market-manc-listings/" |
||||
(format nil "listing-~a.html" (write-to-string counter))))) |
||||
(with-open-file (out-stream |
||||
out-path |
||||
:direction :output |
||||
:if-exists :supersede) |
||||
(format out-stream "~a" item)) |
||||
(incf counter))))))) |
||||
#+end_src |
||||
|
||||
* Create CSV of Listings |
||||
|
||||
#+begin_src lisp :results output |
||||
(with-open-file (out-stream |
||||
#P"working-data/2024-03-01-on-the-market-manc.csv" |
||||
:direction :output |
||||
:if-exists :supersede) |
||||
(let ((row-id 0)) |
||||
(format out-stream "ROW-ID,LOCATION,RENT,URL,RAW-PRICE,RAW-LISTING~%") |
||||
(loop for filepath |
||||
in (directory #P"raw-data/external/2024-03-01-on-the-market-manc-listings/*.html") |
||||
do (with-open-file (in-stream filepath) |
||||
(let* ((doc (plump:parse in-stream)) |
||||
(raw-price (lquery:$ doc ".otm-Price" (text))) |
||||
(cleaned-price (first (cl-ppcre:all-matches-as-strings "[0-9,]+" (aref raw-price 0)))) |
||||
(raw-listing (lquery:$ doc "p" (text))) |
||||
(link (lquery:$ doc "a" (attr :href))) |
||||
(address (lquery:$ doc ".address" (text)))) |
||||
(format t "~a~%" cleaned-price) |
||||
(format out-stream "~d,~a,~d,~a,~a,~a~%" |
||||
row-id |
||||
(str:replace-all "," " " (aref address 0)) |
||||
(str:replace-all "," "" cleaned-price) |
||||
(format nil "https://www.onthemarket.com~a" (aref link 0)) |
||||
(str:replace-all "," "" (aref raw-price 0)) |
||||
(str:replace-all "," " " (aref raw-listing 0))))) |
||||
(incf row-id)))) |
||||
#+end_src |
||||
|
||||
I've gone over the CSV file and all the entries are listed with monthly rent |
||||
rates (and a weekly breakdown next to them). Because of this, I don't need to |
||||
create weekly and monthly breakdowns of the listing, to find the average |
||||
advertised rent. |
||||
|
||||
I manually removed a listing for a garage (listed over £400 pcm). It was only |
||||
one and it was easier to do than code something up. A quick find-and-replace |
||||
search show no entries for 'car park' or 'garage' after that. |
||||
|
||||
The amount of listings in the file is too large to list here. So, here's the |
||||
link to the CSV file, |
||||
|
||||
- [[file:./working-data/2024-03-01-on-the-market-manc.csv]] |
||||
|
||||
* Explore CSV Data for On The Market (2024-03-01) |
||||
|
||||
#+begin_src lisp :session |
||||
(lisp-stat:defdf *otm-manc* |
||||
(lisp-stat:read-csv #P"working-data/2024-03-01-on-the-market-manc.csv")) |
||||
#+end_src |
||||
|
||||
#+RESULTS: |
||||
: #<DATA-FRAME:DATA-FRAME (438 observations of 6 variables)> |
||||
|
||||
The data is in pretty good shape, so don't need to filter out any entries in the |
||||
data-frame. |
||||
|
||||
#+begin_src lisp :session :results file |
||||
(vega:defplot otm-monthly-manc |
||||
`(:title "Rent Rates for Manchester for On The Market (01/03/2024)" |
||||
:description "A breakdown of the rent rates, for On The Markert, in Manchester (01/03/2024)." |
||||
:data ,*otm-manc* |
||||
:width 1200 |
||||
:height 1000 |
||||
:layer #((:mark (:type :bar) |
||||
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0)) |
||||
:y (:field :rent :title "Monthly Rent with Bills Inc. (£)" :type :quantitative) |
||||
:tooltip (:field :rent))) |
||||
(:mark (:type :rule :color "darkorange" :size 3) |
||||
:encoding (:y (:field :rent :type :quantitative :aggregate :average) |
||||
:tooltip (:field :rent :type :quantitative :aggregate :average)))))) |
||||
(vega:write-html otm-monthly-manc "renders/2024-03-01-on-the-market-rent-manc.html") |
||||
#+end_src |
||||
|
||||
#+RESULTS: |
||||
[[file:renders/2024-03-01-on-the-market-rent-manc.html]] |
||||
|
||||
#+begin_src shell |
||||
mv ~/Downloads/visualization.png ./renders/2024-03-01-on-the-market-rent-manc.png |
||||
#+end_src |
||||
|
||||
#+RESULTS: |
||||
|
||||
[[file:./renders/2024-03-01-on-the-market-rent-manc.png]] |
||||
|
||||
#+begin_src lisp :session :results output code raw |
||||
(format t "- Mean Rent: £ ~a~%" (float (lisp-stat:mean *otm-manc*:rent))) |
||||
(format t "- Min. Rent: £ ~d~%" (reduce #'min *otm-manc*:rent)) |
||||
(format t "- Max. Rent: £ ~d" (reduce #'max *otm-manc*:rent)) |
||||
#+end_src |
||||
|
||||
#+RESULTS: |
||||
- Mean Rent: £ 1005.5388 |
||||
- Min. Rent: £ 395 |
||||
- Max. Rent: £ 1250 |
||||
|
||||
* Summary of On The Market |
||||
|
||||
#+begin_src calc :results output |
||||
1005.53 * 12 |
||||
#+end_src |
||||
|
||||
#+RESULTS: |
||||
: 12066.36 |
||||
|
||||
Based on the average rent for On The Market, I would need to make around |
||||
£13,000/yr to cover my living costs. This does not include travel, food, |
||||
clothing, socialising or Income Tax payments. |
||||
|
||||
*NOTE: ON THE MARKET DOESN'T MAKE IT CLEAR IF BILLS ARE INCLUDED*. On top of that, |
||||
I noticed some listing stated they are student accommodation only, even though I |
||||
used the filter option to remove them. This is more the agents abusing the |
||||
system, though. It's not apparent until you start reading the descriptions on |
||||
the listing specific pages. Unfortunately, I can't filter these listings out in |
||||
a practical way, via code. And, there are too many listings to go through |
||||
one-by-one and remove them from the CSV file. |
||||
|
||||
As standard now, I'll add £5,000 to the basic annual rent calculated above. |
||||
|
||||
#+begin_src calc :results output |
||||
12066.36 + 5000 |
||||
#+end_src |
||||
|
||||
#+RESULTS: |
||||
: 17066.36 |
||||
|
||||
Using £17066.36 as the starting point (see [[file:./uk-wage-tax.org][UK Wage and Tax Rates]]), |
||||
|
||||
#+begin_src lisp :results output raw |
||||
(let* ((earning-target 17066.36) |
||||
(p-allow 12570) |
||||
(taxable-income (- earning-target p-allow)) |
||||
(tax-to-pay (* taxable-income 0.2)) |
||||
(total (- earning-target tax-to-pay))) |
||||
(format t "- Annual Target Salary: £~a~%" earning-target) |
||||
(format t "- Part of Salary which is Taxable: £~a~%" taxable-income) |
||||
(format t "- Tax to Pay: £~a~%" tax-to-pay) |
||||
(format t "- Salary After Tax: £~a~%" total)) |
||||
#+end_src |
||||
|
||||
#+RESULTS: |
||||
- Annual Target Salary: £17066.36 |
||||
- Part of Salary which is Taxable: £4496.3594 |
||||
- Tax to Pay: £899.2719 |
||||
- Salary After Tax: £16167.088 |
||||
|
||||
| Time Span | Value After Tax (£) | Mean Rent (£) | |
||||
|-----------------------+---------------------+---------------| |
||||
| Annually | 16167.09 | 1005.53 | |
||||
| Monthly (Before Rent) | 1347.2575 | | |
||||
| Monthly (After Rent) | 341.7275 | | |
||||
| Weekly (After Rent) | 85.431875 | | |
||||
| Daily (After Rent) | 12.204554 | | |
||||
#+TBLFM: @3$2=@-1/12::@4$2=@-1-@-2$+1::@5$2=@-1/4::@6$2=@-1/7 |
||||
|
||||
It looks like using On The Market is the worse route to go down, at the time of |
||||
writing. Having just £12.20/day to spend after paying my rent is the lowest so |
||||
far (2024-03-01 Fri). On top of that On The Market listings not saying if the |
||||
bills are included, they don't make is clear and obvious at least. So, there is |
||||
a strong chance this estimate is way off and the bills will be too much. |
||||
Loading…
Reference in new issue