#+options: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t #+options: broken-links:nil c:nil creator:nil d:(not "LOGBOOK") date:t e:t #+options: email:nil expand-links:t f:t inline:t num:t p:nil pri:nil prop:nil #+options: stat:t tags:t tasks:t tex:t timestamp:t title:t toc:t todo:t |:t #+title: Roomies Manchester #+date: \today #+author: Craig Oates #+email: craig@craigoates.net #+language: en #+select_tags: export #+exclude_tags: noexport #+creator: Emacs 29.1.90 (Org mode 9.7-pre) #+cite_export: * Setup Common Lisp Environment I’ve copied the following code block over from other files. Run it if this is your first file open in the session. #+begin_src lisp :session :results silent (ql:quickload :com.inuoe.jzon) ; JSON parser. (ql:quickload :dexador) ; HTTP requests. (ql:quickload :plump) ; HTML/XML parser. (ql:quickload :lquery) ; HTML/DOM manipulation. (ql:quickload :lparallel) ; Parallel programming. (ql:quickload :cl-ppcre) ; RegEx. library. (ql:quickload :plot/vega) ; Vega plotting library. (ql:quickload :lisp-stat) ; Stat's library. (ql:quickload :data-frame) ; Data frame library eqv. to Python's Numpy. (ql:quickload :str) ; String library, expands on 'string' library. #+end_src * Gather Roomie Data (Manually) - [[https://www.roomies.co.uk/][Roomies]] The website requires you to create an account for full access to the site's listings. With that said, it does provide enough listings to chart the data. The amount of listings is tiny, compared to [[file:./spare-room-manchester.org][Spare Room]] (Manc.) and [[file:./right-move-manchester.org][Right Move]] (Manc.). So, I've downloaded the HTML manually, and going to process that data instead. The search filters I used: - Date: 2024-02-29 - Location: Manchester - Price Range: £0–1200 - Payment Frequency: Month There are only twenty-three listings. I’ve stored the HTML is =raw-data/external/2024-02-29_roomies-manc/=. As is standard behaviour at this point, I won’t be committing that data to the repositories commit history. * Clean Up and Parse Data I'm applying the same tactic as the previous website reviews. I've separated the listings into their own files, reducing the chance of attaching the wrong data to the wrong listings in the CSV files (after parsing the HTML files). #+begin_src shell :results silent mkdir raw-data/external/2024-02-29_roomies-manc-listings/ #+end_src #+begin_src lisp :results silent (let ((counter 0)) (loop for file-path in (directory #P"raw-data/external/2024-02-29_roomies-manc/*.html") do (with-open-file (in-stream file-path) (let* ((doc (plump:parse in-stream)) (listings (lquery:$ doc ".tile" (serialize)))) (loop for item across listings do (let ((out-path (merge-pathnames #P"raw-data/external/2024-02-29_roomies-manc-listings/" (format nil "listing-~a.html" (write-to-string counter))))) (with-open-file (out-stream out-path :direction :output :if-exists :supersede) (format out-stream "~a" item)) (incf counter))))))) #+end_src ** Create CSV Listings #+begin_src lisp :results output raw ;; There is a lot of cleaning of text as the function goes along in this ;; one. The best I can say is take your time when making your way through the ;; code. Essentially, the function has to grab the text, convert it into a text ;; format the code can work with and then strip/replace tokens within the text ;; to make it readable in the CSV file. Yes, this is not my best work, but it ;; works and I don't intend to run this in production for years on end. Get the ;; data, parse it and get out. (let ((filepath #P"working-data/2024-02-29-roomies-manc.csv")) (with-open-file (out-stream filepath :direction :output :if-exists :supersede) (let ((row-id 0)) (format out-stream "ROW-ID,LOCATION,RENT,BILLS-INC,URL~%") (loop for file-path in (directory #P"raw-data/external/2024-02-29_roomies-manc-listings/*.html") do (with-open-file (in-stream file-path) (let* ((doc (plump:parse in-stream)) (listing (lquery:$ doc ".tile" (text))) (price (lquery:$ doc ".text-white" (text))) (cleaned-price (cl-ppcre:regex-replace-all "[\\s\\n]+" (format nil (str:replace-all "," "" (format nil "~s" (aref price 0)))) " ")) (just-price (format nil "~a" (first (cl-ppcre:all-matches-as-strings "\\d+" cleaned-price)))) (bills (format nil "~a" (first (cl-ppcre:all-matches-as-strings "inc." cleaned-price)))) (location (lquery:$ doc ".text-lg" (text))) (cleaned-loc (cl-ppcre:regex-replace-all "[\\s\\n]+" (format nil "~a" (aref location 0)) " ")) (link (lquery:$ doc "a" (attr :href)))) (format out-stream "~d,~a,~d,~a,~a~%" row-id (string-trim " " (str:replace-all "," " " cleaned-loc)) just-price (if (string= "inc." bills) (format nil "Yes") (format nil "No")) (aref link 0))) (incf row-id))))) (format t "[[file:./~a]]" filepath)) #+end_src #+RESULTS: [[file:./working-data/2024-02-29-roomies-manc.csv]] The code above produced a CSV file with a few ~NIL~ results in the ~RENT~ column. These listings are either 'featured' or a 'verified account'. Because the numbers are so small, I've updated them manually. * Explore CSV Data for Roomies (2024-02-29) #+begin_src lisp :session (lisp-stat:defdf *room-manc* (lisp-stat:read-csv #P"working-data/2024-02-29-roomies-manc.csv")) #+end_src #+RESULTS: : # Because the number of listing are so small, compared to [[file:./right-move-manchester.org][Right Move]] and [[file:./spare-room-manchester.org][Spare Room]] (both Manc.) listings, I can output the data here. #+begin_src lisp :session :results output drawer (lisp-stat:print-markdown *room-manc*) #+end_src #+RESULTS: :results: | ROW-ID | LOCATION | RENT | BILLS-INC | URL | |--------+------------------------------------------+------+-----------+----------------------------------------| | 0 | Nobel Way Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/502205 | | 1 | Audenshaw England | 700 | No | https://www.roomies.co.uk/rooms/504567 | | 2 | Openshaw Walk Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/392849 | | 3 | Manchester Road Denton England | 400 | Yes | https://www.roomies.co.uk/rooms/484296 | | 4 | Manchester Street Old Trafford England | 750 | No | https://www.roomies.co.uk/rooms/484350 | | 5 | Beswick Street Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/481551 | | 6 | Mealhouse Lane Atherton England | 650 | No | https://www.roomies.co.uk/rooms/476315 | | 7 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/476586 | | 8 | Old Trafford England | 600 | Yes | https://www.roomies.co.uk/rooms/439451 | | 9 | Droylsden Road Manchester England | 550 | No | https://www.roomies.co.uk/rooms/457342 | | 10 | Audenshaw England | 550 | No | https://www.roomies.co.uk/rooms/453458 | | 11 | Briarfield Road Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/328052 | | 12 | Manchester England | 700 | No | https://www.roomies.co.uk/rooms/472435 | | 13 | Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/406205 | | 14 | Manchester England | 580 | Yes | https://www.roomies.co.uk/rooms/434137 | | 15 | Yates Street Rhodes England | 430 | Yes | https://www.roomies.co.uk/rooms/393284 | | 16 | Stockdale Road Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/344733 | | 17 | Manchester England | 800 | No | https://www.roomies.co.uk/rooms/472436 | | 18 | Talbot Road Manchester England | 450 | Yes | https://www.roomies.co.uk/rooms/267267 | | 19 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/396587 | | 20 | Chelsfield Grove Manchester England | 550 | No | https://www.roomies.co.uk/rooms/499082 | | 21 | Manchester England | 475 | Yes | https://www.roomies.co.uk/rooms/361541 | | 22 | Manchester England | 575 | Yes | https://www.roomies.co.uk/rooms/354296 | | 23 | Manchester England | 650 | Yes | https://www.roomies.co.uk/rooms/489135 | :end: I've assumed the position of excluding the listings which don't include bills in the rent advertised. So, I'm need to create a new data-frame, which filters out the listings which don't include bills. #+begin_src lisp :session (lisp-stat:defdf *room-manc-filt* (lisp-stat:filter-rows *room-manc* '(string= "Yes" bills-inc))) #+end_src #+RESULTS: : # The data with just the listings which include bills in the rent. #+begin_src lisp :session :results output drawer (lisp-stat:print-markdown *room-manc-filt*) #+end_src #+RESULTS: :results: | ROW-ID | LOCATION | RENT | BILLS-INC | URL | |--------+--------------------------------------+------+-----------+----------------------------------------| | 0 | Nobel Way Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/502205 | | 2 | Openshaw Walk Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/392849 | | 3 | Manchester Road Denton England | 400 | Yes | https://www.roomies.co.uk/rooms/484296 | | 5 | Beswick Street Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/481551 | | 7 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/476586 | | 8 | Old Trafford England | 600 | Yes | https://www.roomies.co.uk/rooms/439451 | | 11 | Briarfield Road Manchester England | 700 | Yes | https://www.roomies.co.uk/rooms/328052 | | 13 | Manchester England | 1000 | Yes | https://www.roomies.co.uk/rooms/406205 | | 14 | Manchester England | 580 | Yes | https://www.roomies.co.uk/rooms/434137 | | 15 | Yates Street Rhodes England | 430 | Yes | https://www.roomies.co.uk/rooms/393284 | | 16 | Stockdale Road Manchester England | 750 | Yes | https://www.roomies.co.uk/rooms/344733 | | 18 | Talbot Road Manchester England | 450 | Yes | https://www.roomies.co.uk/rooms/267267 | | 19 | Manchester England | 600 | Yes | https://www.roomies.co.uk/rooms/396587 | | 21 | Manchester England | 475 | Yes | https://www.roomies.co.uk/rooms/361541 | | 22 | Manchester England | 575 | Yes | https://www.roomies.co.uk/rooms/354296 | | 23 | Manchester England | 650 | Yes | https://www.roomies.co.uk/rooms/489135 | :end: #+begin_src lisp :session :results file (vega:defplot roomie-monthly-manc `(:title "Rent Rates for Manchester on Roomies (29/02/2024)" :width 600 :height 600 :data ,*room-manc-filt* :layer #((:mark (:type :bar) :encoding (:x (:field :row-id :title "Assigned Id." :type :nominal :axis ("labelAngle" 0)) :y (:field :rent :title "Monthly Rent with Bills Inc. (£)" :type :quantitative) :tooltip (:field :rent))) (:mark (:type rule :color "darkorange" :size 3) :encoding (:y (:field :rent :type :quantitative :aggregate :average) :tooltip (:field :rent :type :quantitative :aggregate :average)))))) (vega:write-html roomie-monthly-manc "renders/2024-02-29-roomies-rent-manc.html") #+end_src #+RESULTS: [[file:renders/2024-02-29-roomies-rent-manc.html]] [[file:./renders/2024-02-29-roomies-rent-manc.png]] #+begin_src lisp :session :results output code raw (format t "- Mean Rent: £ ~a~%" (float (lisp-stat:mean *room-manc-filt*:rent))) (format t "- Min. Rent: £ ~d~%" (reduce #'min *room-manc-filt*:rent)) (format t "- Max. Rent: £ ~d" (reduce #'max *room-manc-filt*:rent)) #+end_src #+RESULTS: - Mean Rent: £ 641.25 - Min. Rent: £ 400 - Max. Rent: £ 1000 * Summary of Roomies Data #+begin_src calc :results output 641.25 * 12 #+end_src #+RESULTS: : 7695 Based on the average rent (inc. bills) for Roomies, I would need to make around £8,000/yr. to cover my living expenses. This does not include, travel, food, clothing, socialising or income tax payments. In the other Manchester reviews, I've been adding £5,000 on top of the basic rent total. This is so I have a minimum threshold for money I need to make to cover my costs (i.e. to survive). So, adding the £5,000 on top… #+begin_src calc :results output 7695 + 5000 #+end_src #+RESULTS: : 12695 Using £12695 as the starting point (see [[file:./uk-wage-tax.org][UK Wage and Tax Rates]]), #+begin_src lisp :results output raw (let* ((earning-target 12695) (p-allow 12570) (taxable-income (- earning-target p-allow)) (tax-to-pay (* taxable-income 0.2)) (total (- earning-target tax-to-pay))) (format t "- Annual Target Salary: £~a~%" earning-target) (format t "- Part of Salary which is Taxable: £~a~%" taxable-income) (format t "- Tax to Pay: £~a~%" tax-to-pay) (format t "- Salary After Tax: £~a~%" total)) #+end_src #+RESULTS: - Annual Target Salary: £12695 - Part of Salary which is Taxable: £125 - Tax to Pay: £25.0 - Salary After Tax: £12670.0 | Time Span | Value After Tax (£) | Mean Rent (£) | |-----------------------+---------------------+---------------| | Annually | 12670 | 641.25 | | Monthly (Before Rent) | 1055.8333 | | | Monthly (After Rent) | 414.5833 | | | Weekly (After Rent) | 103.64583 | | | Daily (After Rent) | 14.806547 | | #+TBLFM: @3$2=@-1/12::@4$2=@-1-@-2$+1::@5$2=@-1/4::@6$2=@-1/7 The spending limit of £14.80/day falls inline with the other Manchester-based files. Not great but these figures represent the minimum I need to aim for, regarding wages/salaries.