#+options: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t #+options: broken-links:nil c:nil creator:nil d:(not "LOGBOOK") date:t e:t #+options: email:nil expand-links:t f:t inline:t num:t p:nil pri:nil prop:nil #+options: stat:t tags:t tasks:t tex:t timestamp:t title:t toc:t todo:t |:t #+title: Ideal Flatmate Manchester #+date: \today #+author: Craig Oates #+email: craig@craigoates.net #+language: en #+select_tags: export #+exclude_tags: noexport #+creator: Emacs 29.1.90 (Org mode 9.7-pre) #+cite_export: * Gather Ideal Flatmate Data (Manually) - [[https://www.idealflatmate.co.uk/][Ideal Flatmate]] Having had a quick look on the website and did a search with the following filters: - Date: 2024-02-24 Sat - Location: Manchester (City) - Price Range: £0-1200 - Distance: +20 km There are only two pages of results – sixteen listings in total. So, I’ve just saved the HTML manually, from within the browser. Because they are HTML files and come with JavaScript, CSS, images etc., I stored in =raw-data/external/2024-02-24_ideal-flatmate-manc=. These files will not be committed to the repository because I don’t want to clog it up with excess files and images. I just want the rent rates and location data. * Setup Common Lisp Environment You will not need to execute this code block if you've already set up SLIME in another ORG file. This is just in case this is the only file you're working on today, or it's your first file of the day. *Run ~m-x slime~ before running the following code.* And, make note of the ~:session~ attribute. It allows you to use the code in the code block to be use in other code blocks which also use the ~:session~ attribute. #+begin_src lisp :session :results silent (ql:quickload :com.inuoe.jzon) ; JSON parser. (ql:quickload :dexador) ; HTTP requests. (ql:quickload :plump) ; HTML/XML parser. (ql:quickload :lquery) ; HTML/DOM manipulation. (ql:quickload :lparallel) ; Parallel programming. (ql:quickload :cl-ppcre) ; RegEx. library. (ql:quickload :plot/vega) ; Vega plotting library. (ql:quickload :lisp-stat) ; Stat's library. (ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy. (ql:quickload :str) ; String library, expands on 'string' library. #+end_src * Clean Up and Parse Data I'm taking a leaf out of the [[file:./spare-room-manchester.org][Spare Room (Manc)]] book and separating the individual listings into their own files. I've already got code I can quickly adapt to do this and it gives me more confidence around attaching values to the wrong listings. #+begin_src shell :results silent mkdir raw-data/external/2024-02-24_ideal-flatmate-manc-listings/ #+end_src #+begin_src lisp :results silent (let ((counter 0)) (loop for file-path in (directory #P"raw-data/external/2024-02-24_ideal-flatmate-manc/*.html") do (with-open-file (in-stream file-path) (let* ((doc (plump:parse in-stream)) (listings (lquery:$ doc ".card-infos-flex-row" (serialize)))) (loop for item across listings do (let ((out-path (merge-pathnames #P"raw-data/external/2024-02-24_ideal-flatmate-manc-listings/" (format nil "listing-~a.html" (write-to-string counter))))) (with-open-file (out-stream out-path :direction :output :if-exists :supersede) (format out-stream "~a" item)) (incf counter))))))) #+end_src ** Create CSV of Listings #+begin_src lisp :results output raw (let ((filepath #P"working-data/2024-02-24-ideal-flatmate-manc.csv")) (with-open-file (out-stream filepath :direction :output :if-exists :supersede) (let ((row-id 0)) (format out-stream "ROW-ID,RAW-LISTING-INFO,RENT,LOCATION,URL~%") (loop for file-path in (directory #P"raw-data/external/2024-02-24_ideal-flatmate-manc-listings/*.html") do (with-open-file (in-stream file-path) (let* ((doc (plump:parse in-stream)) (listing (lquery:$ doc ".card-infos-left" (text))) (cleaned-price (first (cl-ppcre:all-matches-as-strings "\\d+" (first (str:split "room" (aref listing 0)))))) (cleaned-location (str:replace-all "," "" (second (str:split "room" (aref listing 0))))) (url (lquery:$ doc "a" (attr "href")))) (format out-stream "~d,~s,~d,~a,~a~%" row-id (aref listing 0) cleaned-price cleaned-location (aref url 0)))) (incf row-id)))) (format t "[[file:./working-data/2024-02-24-ideal-flatmate-manc.csv]]")) #+end_src #+RESULTS: [[file:./working-data/2024-02-24-ideal-flatmate-manc.csv]] The amount of listings on Ideal Flatmate is tiny compared to the others. I can basically show all them here. #+begin_src shell :results output raw csvlook working-data/2024-02-24-ideal-flatmate-manc.csv # The output is in markdown format and not org-mode, so you will need to # manually change the section divider, '| --- | --- |' to '|---+---|'. #+end_src #+NAME: im-manc-2024-02-24 #+RESULTS: | ROW-ID | RAW-LISTING-INFO | RENT | LOCATION | URL | |--------+---------------------------------------------------------------------+------+----------------------------------------------+-------------------------------------------------------------------------| | 0 | £690/month per roomChapel Street, Salford M3 5DZ, UK | 690 | Chapel Street Salford M3 5DZ UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113377 | | 1 | £740/month per roomChapel Street, Salford M3 5DZ, UK | 740 | Chapel Street Salford M3 5DZ UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113378 | | 2 | £841 - £842/month per roomMiddlewood Street, Salford, M5 4YW, UK | 841 | Middlewood Street Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120130 | | 3 | £746 - £750/month per roomSalford, M5 4ZF, UK | 746 | Salford M5 4ZF UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id122936 | | 4 | £200/month 100, 100 Lloyd Mansions, Salford M6 6HA, UK | 200 | NIL | https://www.idealflatmate.co.uk/spare-room/salford/property-id122970 | | 5 | £488/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | 488 | Joshua Grange Pluto Cl Salford M6 6HF UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id122929 | | 6 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | 580 | Joshua Grange Pluto Cl Salford M6 6HF UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id123025 | | 7 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK | 480 | Greater Manchester Manchester M31 4HZ 296 UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id122962 | | 8 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | 580 | Joshua Grange Pluto Cl Salford M6 6HF UK | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=242365&pid=123025 | | 9 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK | 480 | Greater Manchester Manchester M31 4HZ 296 UK | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=210168&pid=122962 | | 10 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK | 850 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121033 | | 11 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK | 850 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121032 | | 12 | £956 - £957/month per room7 Symphony Park, Manchester M1 7GB, UK | 956 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121034 | | 13 | £980/month per room7 Symphony Park, Manchester M1 7GB, UK | 980 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121030 | | 14 | £678 - £679/month per roomSalford M5 4YW, UK | 678 | Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120131 | | 15 | £708 - £709/month per roomSalford M5 4YW, UK | 708 | Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120127 | | 16 | £725/month per roomSalford M5 4YW, UK | 725 | Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120128 | | 17 | £775/month per roomMiddlewood Street, Salford, M5 4YW, UK | 775 | Middlewood Street Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120129 | - Min. Rent: £480 - Max. Rent: £956 As you can see, all the entries are listed =/month=. So, I've not bothered to create a /frequency/ column. I'm just going to move forward with that as the default. There are a could of entries in the =URL= column which have valid URL's but point to the incorrect page on the website. I'm just going to leave them in and manually go to their listings page on the website. There isn't enough incorrect URL's to fix this in code. On top of that, I'm more interested in the average rent price per month than the actual locations right now. * Explore CSV Data for Ideal Flatmate (2024-02-24) The listing, in [[im-manc-2024-02-24][the table]] listed above, with the =ROW-ID= of =4=, has rent advertised at =£200/month=. This looked quite a bit off to the other listings. So, I did a quick check and this is a *student-only* listing. Thus, I'm going to remove it from the data, in an attempt to stop the average/mean from being skewed. #+begin_src lisp :session ; This is without the student-only listing removed. (lisp-stat:defdf *im-manc* (lisp-stat:read-csv #P"working-data/2024-02-24-ideal-flatmate-manc.csv")) #+end_src #+RESULTS: : # #+begin_src lisp :session ;; Removed the student listing. (lisp-stat:defdf *im-manc-filt* (lisp-stat:filter-rows *im-manc* '(> rent 201))) #+end_src #+RESULTS: : # #+begin_src lisp :session :results file (vega:defplot monthly-rents `(:title "Rent Rates for Manchester on Ideal Flatmate (24/02/2024)" :width 600 :height 600 :data ,*im-manc-filt* :layer #((:mark (:type :bar) :encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0)) :y (:field :rent :title "Rent (£)" :type :quantitative) :tooltip (:field :rent))) (:mark (:type rule :color "darkorange" :size 3) :encoding (:y (:field :rent :type :quantitative :aggregate :average) :tooltip (:field :rent :type :quantitative :aggregate :average)))))) (vega:write-html monthly-rents "renders/2024-02-24-ideal-flatmate-rent-manc.html") #+end_src #+RESULTS: [[file:renders/2024-02-24-ideal-flatmate-rent-manc.html]] [[file:./renders/2024-02-24-ideal-flatmate-rent-manc.png]] #+begin_src lisp :results output raw (format t "Mean Rent: £~a" (float (lisp-stat:mean *im-manc-filt*:rent))) #+end_src #+RESULTS: Mean Rent: £714.5294 * Summary of Ideal Flatmate Data #+begin_src calc :results output 714.52 * 12 #+end_src #+RESULTS: : 8574.24 Based on the average rent (inc. bill) for Ideal Flatmate, I would need to make about £9000/yr. This is just to cover living expenses. Up to now, I've been adding £5000 on top of that to cover travel, food, clothing and socialising costs. #+begin_src calc :results output 9000 + 5000 #+end_src #+RESULTS: : 14000 #+begin_src calc :result output 5000 / 12 #+end_src #+RESULTS: : 416.666666667 Based on the tax rates (see [[file:./uk-wage-tax.org][UK Wage and Tax Rates]]), the Personal Allowance is £12,570. #+begin_src lisp :results output raw (let* ((earning-target 14000) (p-allow 12570) (taxable-income (- earning-target p-allow)) (tax-to-pay (* taxable-income 0.2)) (total (- earning-target tax-to-pay))) (format t "- Annual Target Salary: £~a~%" earning-target) (format t "- Part of Salary which is Taxable: £~a~%" taxable-income) (format t "- Tax to Pay: £~a~%" tax-to-pay) (format t "- Salary After Tax: £~a~%" total)) #+end_src #+RESULTS: - Annual Target Salary: £14000 - Part of Salary which is Taxable: £1430 - Tax to Pay: £286.0 - Salary After Tax: £13714.0 | Time Span | Value After Tax (£) | Mean Rent (£) | |-----------------------+---------------------+---------------| | Annually | 13714 | 714.52 | | Monthly (Before Rent) | 1142.8333 | | | Monthly (After Rent) | 428.3133 | | | Weekly (After Rent) | 107.07833 | | | Daily (After Rent) | 15.296904 | | #+TBLFM: @3$2=@-1/12::@4$2=@-1-@-2$+1::@5$2=@-1/4::@6$2=@-1/7 The spending limit of £15.29/day is, pretty much, on par with the assessment of the [[file:./spare-room-manchester.org][Spare Room]] listings: not great. And, at the time of writing, I've not taken into account the Income Tax with the Spare Room figures. Granted the difference won't be that different because the annual earnings are too low, for the most part (falling out of the Basic Rate band).