overhaul2024/ideal-flatmate-manchester.org

#+options: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t
#+options: broken-links:nil c:nil creator:nil d:(not "LOGBOOK") date:t e:t
#+options: email:nil expand-links:t f:t inline:t num:t p:nil pri:nil prop:nil
#+options: stat:t tags:t tasks:t tex:t timestamp:t title:t toc:t todo:t |:t
#+title: Ideal Flatmate Manchester
#+date: \today
#+author: Craig Oates
#+email: craig@craigoates.net
#+language: en
#+select_tags: export
#+exclude_tags: noexport
#+creator: Emacs 29.1.90 (Org mode 9.7-pre)
#+cite_export:

* Gather Ideal Flatmate Data (Manually)

- [[https://www.idealflatmate.co.uk/][Ideal Flatmate]]

Having had a quick look on the website and did a search with the following
filters:

- Date: 2024-02-24 Sat
- Location: Manchester (City)
- Price Range: £0-1200
- Distance: +20 km

There are only two pages of results – sixteen listings in total. So, I’ve just
saved the HTML manually, from within the browser. Because they are HTML files
and come with JavaScript, CSS, images etc., I stored in
=raw-data/external/2024-02-24_ideal-flatmate-manc=. These files will not be
committed to the repository because I don’t want to clog it up with excess files
and images. I just want the rent rates and location data.

* Setup Common Lisp Environment

You will not need to execute this code block if you've already set up SLIME in
another ORG file. This is just in case this is the only file you're working on
today, or it's your first file of the day.

*Run ~m-x slime~ before running the following code.* And, make note of the
~:session~ attribute. It allows you to use the code in the code block to be use
in other code blocks which also use the ~:session~ attribute.

#+begin_src lisp :session :results silent
  (ql:quickload :com.inuoe.jzon) ; JSON parser.
  (ql:quickload :dexador)        ; HTTP requests.
  (ql:quickload :plump)          ; HTML/XML parser.
  (ql:quickload :lquery)         ; HTML/DOM manipulation.
  (ql:quickload :lparallel)      ; Parallel programming.
  (ql:quickload :cl-ppcre)       ; RegEx. library.
  (ql:quickload :plot/vega)      ; Vega plotting library.
  (ql:quickload :lisp-stat)      ; Stat's library.
  (ql:quickload :data-frame)     ; Data frame library eqv. to Python's Numpy.
  (ql:quickload :str)            ; String library, expands on 'string' library.
#+end_src

* Clean Up and Parse Data

I'm taking a leaf out of the [[file:./spare-room-manchester.org][Spare Room (Manc)]] book and separating the
individual listings into their own files. I've already got code I can quickly
adapt to do this and it gives me more confidence around attaching values to the
wrong listings.

#+begin_src shell :results silent
  mkdir raw-data/external/2024-02-24_ideal-flatmate-manc-listings/
#+end_src

#+begin_src lisp :results silent
  (let ((counter 0))
    (loop for file-path
          in (directory #P"raw-data/external/2024-02-24_ideal-flatmate-manc/*.html")
          do (with-open-file (in-stream file-path)
               (let* ((doc (plump:parse in-stream))
                      (listings (lquery:$ doc ".card-infos-flex-row" (serialize))))
                 (loop for item across listings
                       do (let ((out-path
                                  (merge-pathnames #P"raw-data/external/2024-02-24_ideal-flatmate-manc-listings/"
                                                   (format nil "listing-~a.html" (write-to-string counter)))))
                            (with-open-file (out-stream
                                             out-path
                                             :direction :output
                                             :if-exists :supersede)
                              (format out-stream "~a" item))
                            (incf counter)))))))
#+end_src

** Create CSV of Listings

#+begin_src lisp :results output raw
  (let ((filepath #P"working-data/2024-02-24-ideal-flatmate-manc.csv"))
    (with-open-file (out-stream
                     filepath
                     :direction :output
                     :if-exists :supersede)
      (let ((row-id 0))
        (format out-stream "ROW-ID,RAW-LISTING-INFO,RENT,LOCATION,URL~%")
        (loop for file-path
                in (directory #P"raw-data/external/2024-02-24_ideal-flatmate-manc-listings/*.html")
              do (with-open-file (in-stream file-path)
                   (let* ((doc (plump:parse in-stream))
                          (listing (lquery:$ doc ".card-infos-left" (text)))
                          (cleaned-price
                            (first (cl-ppcre:all-matches-as-strings
                                    "\\d+"
                                    (first (str:split "room" (aref listing 0))))))
                          (cleaned-location
                            (str:replace-all "," ""
                                             (second (str:split "room" (aref listing 0)))))
                          (url (lquery:$ doc "a" (attr "href"))))
                     (format out-stream "~d,~s,~d,~a,~a~%"
                             row-id
                             (aref listing 0)
                             cleaned-price
                             cleaned-location
                             (aref url 0))))
                 (incf row-id))))
    (format t "[[file:./working-data/2024-02-24-ideal-flatmate-manc.csv]]"))
#+end_src

#+RESULTS:
[[file:./working-data/2024-02-24-ideal-flatmate-manc.csv]]

The amount of listings on Ideal Flatmate is tiny compared to the others. I can
basically show all them here.

#+begin_src shell :results output raw
  csvlook working-data/2024-02-24-ideal-flatmate-manc.csv
  # The output is in markdown format and not org-mode, so you will need to
  # manually change the section divider, '| --- | --- |' to '|---+---|'.
#+end_src

#+NAME: im-manc-2024-02-24
#+RESULTS:
| ROW-ID | RAW-LISTING-INFO                                                    | RENT | LOCATION                                     | URL                                                                     |
|--------+---------------------------------------------------------------------+------+----------------------------------------------+-------------------------------------------------------------------------|
|      0 | £690/month per roomChapel Street, Salford M3 5DZ, UK                |  690 | Chapel Street Salford M3 5DZ UK              | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113377 |
|      1 | £740/month per roomChapel Street, Salford M3 5DZ, UK                |  740 | Chapel Street Salford M3 5DZ UK              | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113378 |
|      2 | £841 - £842/month per roomMiddlewood Street, Salford, M5 4YW, UK    |  841 | Middlewood Street Salford M5 4YW UK          | https://www.idealflatmate.co.uk/spare-room/salford/property-id120130    |
|      3 | £746 - £750/month per roomSalford, M5 4ZF, UK                       |  746 | Salford M5 4ZF UK                            | https://www.idealflatmate.co.uk/spare-room/salford/property-id122936    |
|      4 | £200/month 100, 100 Lloyd Mansions, Salford M6 6HA, UK              |  200 | NIL                                          | https://www.idealflatmate.co.uk/spare-room/salford/property-id122970    |
|      5 | £488/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK      |  488 | Joshua Grange Pluto Cl Salford M6 6HF UK     | https://www.idealflatmate.co.uk/spare-room/salford/property-id122929    |
|      6 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK      |  580 | Joshua Grange Pluto Cl Salford M6 6HF UK     | https://www.idealflatmate.co.uk/spare-room/salford/property-id123025    |
|      7 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK |  480 | Greater Manchester Manchester M31 4HZ 296 UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id122962 |
|      8 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK      |  580 | Joshua Grange Pluto Cl Salford M6 6HF UK     | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=242365&pid=123025   |
|      9 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK |  480 | Greater Manchester Manchester M31 4HZ 296 UK | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=210168&pid=122962   |
|     10 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK           |  850 | 7 Symphony Park Manchester M1 7GB UK         | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121033 |
|     11 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK           |  850 | 7 Symphony Park Manchester M1 7GB UK         | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121032 |
|     12 | £956 - £957/month per room7 Symphony Park, Manchester M1 7GB, UK    |  956 | 7 Symphony Park Manchester M1 7GB UK         | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121034 |
|     13 | £980/month per room7 Symphony Park, Manchester M1 7GB, UK           |  980 | 7 Symphony Park Manchester M1 7GB UK         | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121030 |
|     14 | £678 - £679/month per roomSalford M5 4YW, UK                        |  678 | Salford M5 4YW UK                            | https://www.idealflatmate.co.uk/spare-room/salford/property-id120131    |
|     15 | £708 - £709/month per roomSalford M5 4YW, UK                        |  708 | Salford M5 4YW UK                            | https://www.idealflatmate.co.uk/spare-room/salford/property-id120127    |
|     16 | £725/month per roomSalford M5 4YW, UK                               |  725 | Salford M5 4YW UK                            | https://www.idealflatmate.co.uk/spare-room/salford/property-id120128    |
|     17 | £775/month per roomMiddlewood Street, Salford, M5 4YW, UK           |  775 | Middlewood Street Salford M5 4YW UK          | https://www.idealflatmate.co.uk/spare-room/salford/property-id120129    |

- Min. Rent: £480
- Max. Rent: £956

As you can see, all the entries are listed =/month=. So, I've not bothered to
create a /frequency/ column. I'm just going to move forward with that as the
default.

There are a could of entries in the =URL= column which have valid URL's but
point to the incorrect page on the website. I'm just going to leave them in and
manually go to their listings page on the website. There isn't enough incorrect
URL's to fix this in code. On top of that, I'm more interested in the average
rent price per month than the actual locations right now.

* Explore CSV Data for Ideal Flatmate (2024-02-24)

The listing, in [[im-manc-2024-02-24][the table]] listed above, with the =ROW-ID= of =4=, has rent
advertised at =£200/month=. This looked quite a bit off to the other
listings. So, I did a quick check and this is a *student-only* listing. Thus,
I'm going to remove it from the data, in an attempt to stop the average/mean
from being skewed.

#+begin_src lisp :session
  ; This is without the student-only listing removed.
  (lisp-stat:defdf *im-manc*
      (lisp-stat:read-csv #P"working-data/2024-02-24-ideal-flatmate-manc.csv"))
#+end_src

#+RESULTS:
: #<DATA-FRAME:DATA-FRAME (18 observations of 5 variables)>

#+begin_src lisp :session
  ;; Removed the student listing.
  (lisp-stat:defdf *im-manc-filt*
      (lisp-stat:filter-rows *im-manc* '(> rent 201)))
#+end_src

#+RESULTS:
: #<DATA-FRAME:DATA-FRAME (17 observations of 5 variables)>

#+begin_src lisp :session :results file
  (vega:defplot monthly-rents
    `(:title "Rent Rates for Manchester on Ideal Flatmate (24/02/2024)"
      :width 600
      :height 600
      :data ,*im-manc-filt*
      :layer #((:mark (:type :bar)
               :encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0))
                          :y (:field :rent :title "Rent (£)" :type :quantitative)
                          :tooltip (:field :rent)))
               (:mark (:type rule :color "darkorange" :size 3)
                :encoding (:y (:field :rent :type :quantitative :aggregate :average)
                           :tooltip (:field :rent :type :quantitative :aggregate :average))))))
  (vega:write-html monthly-rents "renders/2024-02-24-ideal-flatmate-rent-manc.html")
#+end_src

#+RESULTS:
[[file:renders/2024-02-24-ideal-flatmate-rent-manc.html]]

[[file:./renders/2024-02-24-ideal-flatmate-rent-manc.png]]

#+begin_src lisp :results output raw
  (format t "Mean Rent: £~a" (float (lisp-stat:mean *im-manc-filt*:rent)))
#+end_src

#+RESULTS:
Mean Rent: £714.5294

* Summary of Ideal Flatmate Data

#+begin_src calc :results output
  714.52 * 12
#+end_src

#+RESULTS:
: 8574.24

Based on the average rent (inc. bill) for Ideal Flatmate, I would need to make
about £9000/yr. This is just to cover living expenses. Up to now, I've been
adding £5000 on top of that to cover travel, food, clothing and socialising
costs.

#+begin_src calc :results output
  9000 + 5000
#+end_src

#+RESULTS:
: 14000

#+begin_src calc :result output
5000 / 12          
#+end_src

#+RESULTS:
: 416.666666667

Based on the tax rates (see [[file:./uk-wage-tax.org][UK Wage and Tax Rates]]), the Personal Allowance is
£12,570.

#+begin_src lisp :results output raw
  (let* ((earning-target 14000)
         (p-allow 12570)
         (taxable-income (- earning-target p-allow))
         (tax-to-pay (* taxable-income 0.2))
         (total (- earning-target tax-to-pay)))
    (format t "- Annual Target Salary: £~a~%" earning-target)
    (format t "- Part of Salary which is Taxable: £~a~%" taxable-income)
    (format t "- Tax to Pay: £~a~%" tax-to-pay)
    (format t "- Salary After Tax: £~a~%" total))
#+end_src

#+RESULTS:
- Annual Target Salary: £14000
- Part of Salary which is Taxable: £1430
- Tax to Pay: £286.0
- Salary After Tax: £13714.0

| Time Span             | Value After Tax (£) | Mean Rent (£) |
|-----------------------+---------------------+---------------|
| Annually              |               13714 |        714.52 |
| Monthly (Before Rent) |           1142.8333 |               |
| Monthly (After Rent)  |            428.3133 |               |
| Weekly (After Rent)   |           107.07833 |               |
| Daily (After Rent)    |           15.296904 |               |
#+TBLFM: @3$2=@-1/12::@4$2=@-1-@-2$+1::@5$2=@-1/4::@6$2=@-1/7

The spending limit of £15.29/day is, pretty much, on par with the assessment of
the [[file:./spare-room-manchester.org][Spare Room]] listings: not great. And, at the time of writing, I've not taken
into account the Income Tax with the Spare Room figures. Granted the difference
won't be that different because the annual earnings are too low, for the most
part (falling out of the Basic Rate band).