11 KiB
Zoopla Manchester
- Setup Common Lisp Environment
- Gather Zoopla Rent Data (Manually)
- Clean Up and Parse Data
- Explore CSV Data for Zoopla (2024-03-18)
- Summary of Zoopla
Setup Common Lisp Environment
I’ve copied the following code block over from other files. Run it if this is your first file open in the session.
(ql:quickload :com.inuoe.jzon) ; JSON parser.
(ql:quickload :dexador) ; HTTP requests.
(ql:quickload :plump) ; HTML/XML parser.
(ql:quickload :lquery) ; HTML/DOM manipulation.
(ql:quickload :lparallel) ; Parallel programming.
(ql:quickload :cl-ppcre) ; RegEx. library.
(ql:quickload :plot/vega) ; Vega plotting library.
(ql:quickload :lisp-stat) ; Stat's library.
(ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy.
(ql:quickload :str) ; String library, expands on 'string' library.Gather Zoopla Rent Data (Manually)
The site produced a single page so I didn't feel the need to scrap the data. I
just saved the page into raw-data/external/2024-03-18-zoopla-rent-manc/. The
data is the actual HTML and CSS, so the files will not be committed to the
repository, to match the established behaviour of the other sites I've already
looked at.
The filters I used:
- Date: 2024-03-18
- Location: Manchester City Centre, Greater Manchester
- Radius: 'This area only'
- Price Range: £400–£1,300 pcm
- Only monthly rents listed
Clean Up and Parse Data
Going to separate the listings out into their own files, to match the established behaviour with other sites I've looked at.
mkdir raw-data/external/2024-03-18-zoopla-rent-manc-listings//craig.oates/overhaul2024/src/branch/master/raw-data/external/2024-03-18-zoopla-rent-manc-listings
(let ((counter 0))
(loop for file-path
in (directory #P"raw-data/external/2024-03-18-zoopla-rent-manc/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
;; .listing-info
(listings (lquery:$ doc ".dkr2t82" (serialize))))
(loop for item across listings
do (let ((out-path
(merge-pathnames
#P"raw-data/external/2024-03-18-zoopla-rent-manc-listings/"
(format nil "listing-~a.html" (write-to-string counter)))))
(with-open-file (out-stream
out-path
:direction :output
:if-exists :supersede)
(format out-stream "~a~%" item))
(incf counter))))))) COUNT=$(ls -1 "raw-data/external/2024-03-18-zoopla-rent-manc-listings/" | wc -l)
echo "There are $COUNT files in the directory."There are 14 files in the directory.
There are some listings which are student-only. Unfortunately, there doesn't seem to be a filter for removing student-only listings. So, I'll need to remove them manually at little later down the line. The total amount of listing is small enough for it to not be a problem.
Create CSV of Listings
(let ((row-id 0)
(out-file #P"working-data/2024-03-18-zoopla-manc.csv"))
(with-open-file (out-stream
out-file
:direction :output
:if-exists :supersede)
(format out-stream "ROW-ID,RENT,LOCATION,URL,DESCRIPTION~%")
(loop for input
in (directory #P"raw-data/external/2024-03-18-zoopla-rent-manc-listings/*.html")
do (with-open-file (in-stream input)
(let* ((doc (plump:parse in-stream))
(price (lquery:$ doc "._64if862._194zg6t6" (text)))
(description (lquery:$ doc ".m6hnz63._194zg6t9" (text)))
;; e.g. Transform '£1,250 per month' to '1250'.
(cleaned-price
(first
(cl-ppcre:all-matches-as-strings
"\\d+"
(cl-ppcre:regex-replace-all
"[\\s\\n]+"
(format nil "~s" (str:replace-all "," "" (aref price 0)))
" "))))
(location (lquery:$ doc "address" (text)))
(link (lquery:$ doc "a" (attr :href))))
(format out-stream "~d,~d,~a,~a,~a~%"
row-id
cleaned-price
(str:replace-all "," " " (aref location 0))
(cl-ppcre:regex-replace "\\?search.*" (aref link 0) "")
(str:replace-all "," " " (aref description 0)))))
(incf row-id)))
(format t "[[file:./~a]]" out-file))/craig.oates/overhaul2024/src/branch/master/working-data/2024-03-18-zoopla-manc.csv
I've manually removed the student-only listing, since creating the CSV file. File is left with 5 listings.
Explore CSV Data for Zoopla (2024-03-18)
(lisp-stat:defdf *zoop-man*
(lisp-stat:read-csv #P"working-data/2024-03-18-zoopla-manc.csv"))#<DATA-FRAME:DATA-FRAME (5 observations of 5 variables)>
The total amount of listings is tiny, so I can list them out here.
(lisp-stat:print-markdown *zoop-man*)| ROW-ID | RENT | LOCATION | URL | DESCRIPTION |
|---|---|---|---|---|
| 2 | 1150 | Arches Whitworth Street West Manchester M1 | https://www.zoopla.co.uk/to-rent/details/65630906/ | 3 month agreements only - available 23rd March A true Manchester gem minutes from the Northern Quarter Canal Street and Piccadilly. This is … |
| 3 | 1150 | Water Street Manchester M3 | https://www.zoopla.co.uk/to-rent/details/66713818/ | 1-6 months Tenancy bills included no couples due to regulations**Perfect for Contractors & Work Teams****Perfect for Business … |
| 5 | 1250 | Arches Whitworth Street West Manchester M1 | https://www.zoopla.co.uk/to-rent/details/66338812/ | 3 month agreements only - available 23rd March A true Manchester gem minutes from the Northern Quarter Canal Street and Piccadilly. This is … |
| 8 | 1050 | Little Lever Street Manchester Greater Manchester M1 | https://www.zoopla.co.uk/to-rent/details/65042494/ | * available from 19/04/2024 * * all bills inclusive in rent * Reeds Rains are delighted to market this fantastic furnished studio apartment … |
| 11 | 1089 | Cross Street Manchester M2 | https://www.zoopla.co.uk/to-rent/details/66673111/ | Listing code 692841. Our contemporary studio apartments at 25 Cross Street are thoughtfully designed to a high standard.Featuring a comfy double … |
(vega:defplot zoopla-monthly-manc
`(:title "Rent Rates for Manchester on Zoople (18/03/2024)"
:width 600
:height 600
:data ,*zoop-man*
:layer #((:mark (:type :bar)
:encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0))
:y (:field :rent :title "Monthly Rent with Bills Inc. (£)" :type :quantitative)
:tooltip (:field :rent)))
(:mark (:type rule :color "darkorange" :size 3)
:encoding (:y (:field :rent :type :quantitative :aggregate :average)
:tooltip (:field :rent :type :quantitative :aggregate :average))))))
(vega:write-html zoopla-monthly-manc "renders/2024-03-18-zoopla-rent-manc.html")/craig.oates/overhaul2024/src/branch/master/renders/2024-03-18-zoopla-rent-manc.html
mv ~/Downloads/visualization.png ./renders/2024-03-18-zoopla-rent-manc.png
(format t "- Mean Rent: £ ~a~%" (float (lisp-stat:mean *zoop-man*:rent)))
(format t "- Min. Rent: £ ~d~%" (reduce #'min *zoop-man*:rent))
(format t "- Max. Rent: £ ~d" (reduce #'max *zoop-man*:rent))- Mean Rent: £ 1137.8
- Min. Rent: £ 1050
- Max. Rent: £ 1250
1137.8 * 1213653.6
Summary of Zoopla
Based on the average rent for Zoopla, I would need to make around £14,000/yr to cover my living costs. This doesn't include travel, food, clothing, socialising or Income Tax Payments.
Most listing were aimed at students and the number of listings was tiny, especially when compared to Spareroom and Open Rent.
To get an idea of what I need for ’survival mode’, I'll add £5,000 onto the total above and apply the usual calculations, established in the other files.
13653.6 + 500018653.6
Using £18,653.6 as the starting point (see UK Wage and Tax Rated,
(let* ((earning-target 18653.6)
(p-allow 12570)
(taxable-income (- earning-target p-allow))
(tax-to-pay (* taxable-income 0.2))
(total (- earning-target tax-to-pay)))
(format t "- Annual Target Salary: £~a~%" earning-target)
(format t "- Part of Salary which is Taxable: £~a~%" taxable-income)
(format t "- Tax to Pay: £~a~%" tax-to-pay)
(format t "- Salary After Tax: £~a~%" total))- Annual Target Salary: £18653.6
- Part of Salary which is Taxable: £6083.5996
- Tax to Pay: £1216.72
- Salary After Tax: £17436.879
| Time Span | Value After Tax (£) | Mean Rent (£) |
|---|---|---|
| Annually | 17436.88 | 1137.8 |
| Monthly (Before Rent) | 1453.0733 | |
| Monthly (After Rent) | 315.2733 | |
| Weekly (After Rent) | 78.818325 | |
| Daily (After Rent) | 11.259761 |
So, the spending limit is £11.26/day.