10 KiB
On The Market Manchester
- Setup Common Lisp Environment
- Gather On The Market Data
- Clean Up and Parse Data
- Create CSV of Listings
- Explore CSV Data for On The Market (2024-03-01)
- Summary of On The Market
Setup Common Lisp Environment
I’ve copied the following code block over from other files. Run it if this is your first file open in the session.
(ql:quickload :com.inuoe.jzon) ; JSON parser.
(ql:quickload :dexador) ; HTTP requests.
(ql:quickload :plump) ; HTML/XML parser.
(ql:quickload :lquery) ; HTML/DOM manipulation.
(ql:quickload :lparallel) ; Parallel programming.
(ql:quickload :cl-ppcre) ; RegEx. library.
(ql:quickload :plot/vega) ; Vega plotting library.
(ql:quickload :lisp-stat) ; Stat's library.
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
(ql:quickload :str) ; String library, expands on 'string' library.Gather On The Market Data
Having had a quick look on the website, it says there are 438 results matching the following criteria,
- Location: Manchester, Greater Manchester
- To rent
- Max: £1,2500 pcm
- Min: £300 pcm (to filter out car parks)
- No student accommodation
There are too many results to grab the data manually, so I'm gonna have to
scrape it with some code. The website is paginated and the last page for these
results is page=18, as stated in the URL (19 pages but starts from 0).
cd raw-data/external
DIRECTORY="$(date '+%Y-%m-%d')-on-the-market-manc"
mkdir $DIRECTORY
for PAGE in {0..18}
do
curl -o "$DIRECTORY/on-the-market-$PAGE.html" \
"https://www.onthemarket.com/to-rent/property/manchester/?direction=asc&max-price=1250&min-price=300&page=$PAGE&sort-field=price&student=false"
sleep 5
done
# Change back to the project's root directory, so I don't call code whilst still
# in this directory.
cd ../../Clean Up and Parse Data
mkdir raw-data/external/2024-03-01-on-the-market-manc-listings/ (let ((counter 0))
(loop for file-path
in (directory #P"raw-data/external/2024-03-01-on-the-market-manc/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listings (lquery:$ doc ".otm-PropertyCardInfo" (serialize))))
(loop for item across listings
do (let ((out-path
(merge-pathnames "raw-data/external/2024-03-01-on-the-market-manc-listings/"
(format nil "listing-~a.html" (write-to-string counter)))))
(with-open-file (out-stream
out-path
:direction :output
:if-exists :supersede)
(format out-stream "~a" item))
(incf counter)))))))Create CSV of Listings
(with-open-file (out-stream
#P"working-data/2024-03-01-on-the-market-manc.csv"
:direction :output
:if-exists :supersede)
(let ((row-id 0))
(format out-stream "ROW-ID,LOCATION,RENT,URL,RAW-PRICE,RAW-LISTING~%")
(loop for filepath
in (directory #P"raw-data/external/2024-03-01-on-the-market-manc-listings/*.html")
do (with-open-file (in-stream filepath)
(let* ((doc (plump:parse in-stream))
(raw-price (lquery:$ doc ".otm-Price" (text)))
(cleaned-price (first (cl-ppcre:all-matches-as-strings "[0-9,]+" (aref raw-price 0))))
(raw-listing (lquery:$ doc "p" (text)))
(link (lquery:$ doc "a" (attr :href)))
(address (lquery:$ doc ".address" (text))))
(format t "~a~%" cleaned-price)
(format out-stream "~d,~a,~d,~a,~a,~a~%"
row-id
(str:replace-all "," " " (aref address 0))
(str:replace-all "," "" cleaned-price)
(format nil "https://www.onthemarket.com~a" (aref link 0))
(str:replace-all "," "" (aref raw-price 0))
(str:replace-all "," " " (aref raw-listing 0)))))
(incf row-id))))I've gone over the CSV file and all the entries are listed with monthly rent rates (and a weekly breakdown next to them). Because of this, I don't need to create weekly and monthly breakdowns of the listing, to find the average advertised rent.
I manually removed a listing for a garage (listed over £400 pcm). It was only one and it was easier to do than code something up. A quick find-and-replace search show no entries for 'car park' or 'garage' after that.
The amount of listings in the file is too large to list here. So, here's the link to the CSV file,
Explore CSV Data for On The Market (2024-03-01)
(lisp-stat:defdf *otm-manc*
(lisp-stat:read-csv #P"working-data/2024-03-01-on-the-market-manc.csv"))#<DATA-FRAME:DATA-FRAME (438 observations of 6 variables)>
The data is in pretty good shape, so don't need to filter out any entries in the data-frame.
(vega:defplot otm-monthly-manc
`(:title "Rent Rates for Manchester for On The Market (01/03/2024)"
:description "A breakdown of the rent rates, for On The Markert, in Manchester (01/03/2024)."
:data ,*otm-manc*
:width 1200
:height 1000
:layer #((:mark (:type :bar)
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0))
:y (:field :rent :title "Monthly Rent with Bills Inc. (£)" :type :quantitative)
:tooltip (:field :rent)))
(:mark (:type :rule :color "darkorange" :size 3)
:encoding (:y (:field :rent :type :quantitative :aggregate :average)
:tooltip (:field :rent :type :quantitative :aggregate :average))))))
(vega:write-html otm-monthly-manc "renders/2024-03-01-on-the-market-rent-manc.html")/craig.oates/overhaul2024/src/branch/master/renders/2024-03-01-on-the-market-rent-manc.html
mv ~/Downloads/visualization.png ./renders/2024-03-01-on-the-market-rent-manc.png
(format t "- Mean Rent: £ ~a~%" (float (lisp-stat:mean *otm-manc*:rent)))
(format t "- Min. Rent: £ ~d~%" (reduce #'min *otm-manc*:rent))
(format t "- Max. Rent: £ ~d" (reduce #'max *otm-manc*:rent))- Mean Rent: £ 1005.5388
- Min. Rent: £ 395
- Max. Rent: £ 1250
Summary of On The Market
1005.53 * 1212066.36
Based on the average rent for On The Market, I would need to make around £13,000/yr to cover my living costs. This does not include travel, food, clothing, socialising or Income Tax payments.
NOTE: ON THE MARKET DOESN'T MAKE IT CLEAR IF BILLS ARE INCLUDED. On top of that, I noticed some listing stated they are student accommodation only, even though I used the filter option to remove them. This is more the agents abusing the system, though. It's not apparent until you start reading the descriptions on the listing specific pages. Unfortunately, I can't filter these listings out in a practical way, via code. And, there are too many listings to go through one-by-one and remove them from the CSV file.
As standard now, I'll add £5,000 to the basic annual rent calculated above.
12066.36 + 500017066.36
Using £17066.36 as the starting point (see UK Wage and Tax Rates),
(let* ((earning-target 17066.36)
(p-allow 12570)
(taxable-income (- earning-target p-allow))
(tax-to-pay (* taxable-income 0.2))
(total (- earning-target tax-to-pay)))
(format t "- Annual Target Salary: £~a~%" earning-target)
(format t "- Part of Salary which is Taxable: £~a~%" taxable-income)
(format t "- Tax to Pay: £~a~%" tax-to-pay)
(format t "- Salary After Tax: £~a~%" total))- Annual Target Salary: £17066.36
- Part of Salary which is Taxable: £4496.3594
- Tax to Pay: £899.2719
- Salary After Tax: £16167.088
| Time Span | Value After Tax (£) | Mean Rent (£) |
|---|---|---|
| Annually | 16167.09 | 1005.53 |
| Monthly (Before Rent) | 1347.2575 | |
| Monthly (After Rent) | 341.7275 | |
| Weekly (After Rent) | 85.431875 | |
| Daily (After Rent) | 12.204554 |
It looks like using On The Market is the worse route to go down, at the time of writing. Having just £12.20/day to spend after paying my rent is the lowest so far (2024-03-01 Fri). On top of that On The Market listings not saying if the bills are included, they don't make is clear and obvious at least. So, there is a strong chance this estimate is way off and the bills will be too much.