You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
244 lines
9.6 KiB
244 lines
9.6 KiB
3 months ago
|
#+title: Overhaul 2024 Journal
|
||
4 months ago
|
#+options: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t
|
||
|
#+options: broken-links:nil c:nil creator:nil d:(not "LOGBOOK") date:t e:t
|
||
|
#+options: email:nil expand-links:t f:t inline:t num:t p:nil pri:nil prop:nil
|
||
|
#+options: stat:t tags:t tasks:t tex:t timestamp:t title:t toc:t todo:t |:t
|
||
|
#+date: \today
|
||
|
#+author: Craig Oates
|
||
|
#+email: craig@craigoates.net
|
||
|
#+language: en
|
||
|
#+select_tags: export
|
||
|
#+exclude_tags: noexport
|
||
|
#+creator: Emacs 29.1.90 (Org mode 9.7-pre)
|
||
|
#+cite_export:
|
||
|
|
||
3 months ago
|
* Setup Directories
|
||
4 months ago
|
|
||
|
This just sets up the work environments directories. You should only need to run
|
||
|
this the first time you work your way thought this file.
|
||
|
|
||
|
#+begin_src shell :results output
|
||
|
mkdir raw-data renders working-data
|
||
|
#+end_src
|
||
|
|
||
3 months ago
|
* Setup Python Virtual Environment (Properties Drawer)
|
||
4 months ago
|
:PROPERTIES:
|
||
|
:header-args:shell: :exports code
|
||
|
:header-args:elisp: :exports code
|
||
|
:header-args:ipython: :exports both
|
||
|
:header-args: :eval never-export
|
||
|
:END:
|
||
|
|
||
|
I've got some code [[https://git.abbether.net/craig.oates/literate-charting/src/branch/master/charting.org][here]] which has information about setting up Python to be used
|
||
|
in a literate programming context with org-mode.
|
||
|
|
||
|
#+begin_src shell
|
||
|
python3 -m venv venv
|
||
|
# To activate the virtual environment in your CLI...
|
||
|
source venv/bin/activate
|
||
|
#+end_src
|
||
|
|
||
|
#+begin_src shell
|
||
|
# Should only need this on first run.
|
||
|
touch requirements.txt
|
||
|
#+end_src
|
||
|
|
||
|
#+begin_src shell
|
||
|
# To activate the virtual environment in your CLI...
|
||
|
source venv/bin/activate
|
||
|
#+end_src
|
||
|
|
||
|
#+begin_src shell :results code
|
||
|
# To check results.
|
||
|
tree -L 1
|
||
|
#+end_src
|
||
|
|
||
|
#+begin_src elisp :results silent
|
||
|
;; Activate at start of coding session...
|
||
|
;; Change path to where you have saved this repository.
|
||
|
(pyvenv-activate "~/dev-shed/overhaul2024/venv")
|
||
|
(pyvenv-mode)
|
||
|
#+end_src
|
||
|
|
||
|
You can, also, do this via =m-x pyvenv-activate=. You should be able to put
|
||
|
~venv~ in the prompt and be good to go. Use =m-x pyvenv-mode= to see the
|
||
|
currently active virtual-environment in the mode-line.
|
||
|
|
||
|
#+begin_src shell :results silent
|
||
|
# Snippets to install and store Python packages (in venv).
|
||
|
# You'll probably be better running these at the CLI.
|
||
|
pip freeze > requirements.txt
|
||
|
pip install -r requirements.txt
|
||
|
#+end_src
|
||
|
|
||
3 months ago
|
* Scrape Right Move Data Manchester (2024-02-18 Sun)
|
||
4 months ago
|
|
||
|
Found some code at [[https://scrapfly.io/blog/how-to-scrape-rightmove/#full-scraper-code][ScrapFly]] for scraping data from [[https://www.rightmove.co.uk][Right Move]]. I put it back
|
||
|
together (in was in code blocks for the purpose of the blog post) and modified
|
||
|
it to what I need. Ideally, I would have the code within this file but the code
|
||
|
is already written and it easier to just run the code as a script file (from
|
||
|
here). *You will need to change the ~location~ variable in the ~run~ function to
|
||
|
search and scrape data from other parts of the country.* The data is stored in
|
||
|
=raw-data/=, as a JSON file, and will require further parsing and processing.
|
||
|
|
||
|
#+begin_src shell :results silent
|
||
|
python rightmove.py
|
||
|
#+end_src
|
||
|
|
||
3 months ago
|
* Setup Common Lisp Environment
|
||
4 months ago
|
|
||
|
*Run ~m-x slime~ before running the following code.* And, make note of the
|
||
|
~:session~ attribute. It allows you to use the code in the code block to be use
|
||
|
in other code blocks which also use the ~:session~ attribute.
|
||
|
|
||
|
#+begin_src lisp :session :results silent
|
||
|
(ql:quickload :com.inuoe.jzon)
|
||
|
(ql:quickload :plot/vega)
|
||
|
(ql:quickload :lisp-stat)
|
||
|
(ql:quickload :data-frame)
|
||
3 months ago
|
(ql:quickload :str)
|
||
4 months ago
|
#+end_src
|
||
|
|
||
3 months ago
|
* Convert Right Move JSON Data (2024-02-18 Sun) for Manchester to CSV File
|
||
4 months ago
|
|
||
|
This code goes through the JSON file, returned by Right Move, and creates a CSV
|
||
|
file of the data I'm most interested in. It's, also, prep. work for using Lisp Stat.
|
||
|
|
||
|
#+begin_src lisp :results silent :session
|
||
3 months ago
|
;; Adjust path to match the file you want to process.
|
||
|
(let ((data (com.inuoe.jzon:parse
|
||
|
#P"raw-data/2024-02-18_21-48-46_right-move-manchester.json")))
|
||
|
(with-open-file (stream
|
||
|
#P"working-data/2024-02-18_21-48-46_right-move-manchester.csv"
|
||
|
:direction :output
|
||
|
:if-exists :supersede)
|
||
|
(format stream "ID, Price, Frequency, Bedrooms, Bathrooms, Display Address, Students, Latitude, Longitude, URL~%")
|
||
|
(loop for homes across data
|
||
|
do (format stream "~a, ~a, ~a, ~a, ~a, ~s, ~a, ~a, ~a, https://www.rightmove.co.uk/properties/~a~%"
|
||
|
(gethash "id" homes)
|
||
|
(gethash "amount" (gethash "price" homes))
|
||
|
(gethash "frequency" (gethash "price" homes))
|
||
|
(gethash "bedrooms" homes)
|
||
|
(gethash "bathrooms" homes)
|
||
|
(gethash "displayAddress" homes)
|
||
|
(gethash "students" homes)
|
||
|
(gethash "latitude" (gethash "location" homes))
|
||
|
(gethash "longitude" (gethash "location" homes))
|
||
|
(gethash "id" homes)))))
|
||
4 months ago
|
#+end_src
|
||
|
|
||
3 months ago
|
* Explore CSV Data for Right Move Manchester (2024-02-18)
|
||
|
|
||
4 months ago
|
#+begin_src lisp :results silent :session
|
||
3 months ago
|
(defvar *rm-manchester*
|
||
4 months ago
|
(lisp-stat:read-csv #P"working-data/2024-02-18_21-48-46_right-move-manchester.csv")
|
||
|
"Data from the CSV file (after it has been processed from the JSON file).")
|
||
|
(lisp-stat:defdf *rm-manc-df* *rm-manchester*)
|
||
|
#+end_src
|
||
|
|
||
3 months ago
|
Having had a quick look at the CSV file, I noticed some rent prices above
|
||
|
£10,000 a month. This is just way outside my budget, so I could do with
|
||
|
filtering them out. There are, also, some entries which are car parks, so I'll
|
||
|
need to filter them out as well. I could, also, do with separating out the
|
||
|
weekly and monthly rent prices so I look at them fairly. Filtering out the
|
||
|
student accommodation is something I should do, as well. I'm not a student
|
||
|
anymore, and haven't been for quite a long time.
|
||
|
|
||
|
#+begin_src lisp :session :results silent
|
||
|
(lisp-stat:write-csv
|
||
|
(lisp-stat:filter-rows *rm-manc-df* '(and
|
||
|
(string= "weekly" frequency)
|
||
|
(string= "NIL" students)))
|
||
|
#P"working-data/2024-02-18_21-48-46_right-move-manchester-weekly.csv"
|
||
|
:add-first-row t)
|
||
4 months ago
|
#+end_src
|
||
|
|
||
3 months ago
|
#+begin_src lisp :session :results silent
|
||
|
(lisp-stat:write-csv
|
||
|
(lisp-stat:filter-rows *rm-manc-df*
|
||
|
'(and
|
||
|
(string= "monthly" frequency)
|
||
|
(> 1500 price)
|
||
|
(string= "NIL" students)
|
||
|
;; Filtering for the car park entries.
|
||
|
(< 275 price)
|
||
|
(not (str:contains? "car " display-address :ignore-case t))
|
||
|
(not (str:contains? "park " display-address :ignore-case t))
|
||
|
(not (str:contains? "parking " display-address :ignore-case t))))
|
||
|
#P"working-data/2024-02-18_21-48-46_right-move-manchester-monthly.csv"
|
||
|
:add-first-row t)
|
||
|
#+end_src
|
||
4 months ago
|
|
||
3 months ago
|
I've stored the /cleaned/ results in separate CSV files because I will need to
|
||
|
work through them away from here – in terms of reviewing the listing on the
|
||
|
Right Move website. Having said that, I am going to do a little bit of /summary/
|
||
|
work so I can see what I need to aim for salary-wise if I’m to get a new job.
|
||
4 months ago
|
|
||
3 months ago
|
#+begin_src lisp :session :results silent
|
||
|
(lisp-stat:defdf *rm-manc-filt-df*
|
||
|
(lisp-stat:read-csv #P"working-data/2024-02-18_21-48-46_right-move-manchester-monthly.csv"))
|
||
|
#+end_src
|
||
4 months ago
|
|
||
3 months ago
|
#+begin_src lisp :session :results drawer
|
||
|
(lisp-stat:summarize-column '*rm-manc-filt-df*:price)
|
||
4 months ago
|
#+end_src
|
||
|
|
||
|
#+RESULTS:
|
||
|
:results:
|
||
3 months ago
|
121 reals, min=945, q25=1098.75, q50=1246.4286, q75=1358.9375, max=1495
|
||
|
:end:
|
||
|
|
||
|
- Monthly Average: £1236.12 (done in CSV file outside of this file)
|
||
|
|
||
|
The /weekly/ version of the Right Move CSV file only has six entries. I'm not
|
||
|
going to bother using Lisp-Stat here. I'm just going to summarise manually.
|
||
|
|
||
|
| Weekly (£) | Monthly (£) | Notes |
|
||
|
|------------+-------------+---------------|
|
||
|
| 348 | 1392 | |
|
||
|
| 238 | 952 | |
|
||
|
| 592 | 2368 | Out of budget |
|
||
|
| 458 | 1832 | Out of budget |
|
||
|
| 458 | 1832 | Out of budget |
|
||
|
| 443 | 1772 | Out of budget |
|
||
|
#+TBLFM: @2$2=4*$1::@3$2=4*$1::@4$2=4*$1::@5$2=4*$1::@6$2=4*$1::@7$2=4*$1
|
||
|
|
||
|
Looks like I'm keeping only two of the weekly entries. Deleted the 'Out of
|
||
|
budget' entries from
|
||
|
=working-data/2024-02-18_21-48-46_right-move-manchester-weekly.csv= file.
|
||
|
|
||
|
Combining the monthly and week averages to get the average rent price for places
|
||
|
to live in Manchester, via Right Move,
|
||
|
|
||
|
#+begin_src lisp :results output raw
|
||
|
(let* ((weekly-avg (/ (+ 1392 952) 2))
|
||
|
(wk-mnt-avg (/ (+ 1236 weekly-avg) 2)))
|
||
|
(format t "Weekly rent average with Right Move (converted to monthly): £~a~%" weekly-avg)
|
||
|
(format t "OVERALL monthly rent for Manchester with Right Move: £~a~%" wk-mnt-avg))
|
||
|
#+end_src
|
||
4 months ago
|
|
||
3 months ago
|
#+RESULTS:
|
||
|
Weekly rent average with Right Move (converted to monthly): £1172
|
||
|
OVERALL monthly rent for Manchester with Right Move: £1204
|
||
4 months ago
|
|
||
3 months ago
|
So, to sum up the properties on Right Move,
|
||
|
|
||
|
| Source | Min. (£) | Max. (£) | Avg. (£) |
|
||
|
|------------+----------+----------+----------|
|
||
|
| Right Move | 945 | 1495 | 1204 |
|
||
|
|
||
|
I formatted the =Annual= values manually. Kept the table formulas for reference.
|
||
|
|
||
|
| Quantity Property | Annual |
|
||
|
|-------------------+---------|
|
||
|
| Minimum | £11,340 |
|
||
|
| Maximum | £17,940 |
|
||
|
| Average | £14,448 |
|
||
|
#+TBLFM: @2$2=945*12::@3$2=1495*12::@4$2=1204*12
|
||
|
|
||
|
** Summary of Right Move Data
|
||
|
|
||
|
I basically need to earn around £15,000/yr just to cover my living costs. This
|
||
3 months ago
|
does not include food, clothes, travel and socialising. Realistically, I'm
|
||
3 months ago
|
looking at aiming for £20,000/yr if I go this route.
|