craig.oates
/
overhaul2024
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Browse Source

finish code for creating a CSV file for Ideal Flatmate.

master
Craig Oates 2 months ago
parent
31c689ef6b
commit
6857f75ad0
1 changed files with 53 additions and 31 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 84
      ideal-flatmate-manchester.org

84
ideal-flatmate-manchester.org
Unescape View File

@ -84,48 +84,70 @@ wrong listings.
(incf counter)))))))
#+end_src
** TODO Create CSV of Listings
** Create CSV of Listings
Need to come back to this and finish it off. Left it in a mess because it's the
end of the day (at time of commit) and need to get some sleep.
#+begin_src lisp :results output raw
;; (with-open-file (out-stream
;; #P"working-data/2024-02-24-ideal-flatmate-manc.csv"
;; :direction :output
;; :if-exists :supersede)
;; (format out-stream "ROW-ID, OTHER STUFF")
#+begin_src lisp :results silent
(with-open-file (out-stream
#P"working-data/2024-02-24-ideal-flatmate-manc.csv"
:direction :output
:if-exists :supersede)
(let ((row-id 0))
(format t "|ROW-ID|LISTING-INFO|URL|~%")
(format t "|-|-|-|~%")
(format out-stream "ROW-ID,RAW-LISTING-INFO,RENT (£),LOCATION,URL~%")
(loop for file-path
in (directory #P"raw-data/external/2024-02-24_ideal-flatmate-manc-listings/*.html")
do (with-open-file (in-stream file-path)
(let* ((doc (plump:parse in-stream))
(listing (lquery:$ doc ".card-infos-left" (text)))
(cleaned-price (first (cl-ppcre:all-matches-as-strings "\\d+" (first (str:split "room" (aref listing 0))))))
(cleaned-location (str:replace-all "," "" (second (str:split "room" (aref listing 0)))))
(url (lquery:$ doc "a" (attr "href"))))
(format t "|~a|~a|~a|~%" row-id (aref listing 0) (aref url 0))))
(incf row-id)))
(format out-stream "~a,~s,~a,~a,~a~%"
row-id
(aref listing 0)
cleaned-price
cleaned-location
(aref url 0))))
(incf row-id))))
#+end_src
The amount of listings on Ideal Flatmate is tiny compared to the others. I can
basically show all them here.
#+begin_src shell :results output raw
csvlook working-data/2024-02-24-ideal-flatmate-manc.csv
#+end_src
#+RESULTS:
| ROW-ID | LISTING-INFO | URL |
|--------+---------------------------------------------------------------------+-------------------------------------------------------------------------|
| 0 | £690/month per roomChapel Street, Salford M3 5DZ, UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113377 |
| 1 | £740/month per roomChapel Street, Salford M3 5DZ, UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113378 |
| 2 | £841 - £842/month per roomMiddlewood Street, Salford, M5 4YW, UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120130 |
| 3 | £746 - £750/month per roomSalford, M5 4ZF, UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id122936 |
| 4 | £200/month 100, 100 Lloyd Mansions, Salford M6 6HA, UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id122970 |
| 5 | £488/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id122929 |
| 6 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id123025 |
| 7 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id122962 |
| 8 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=242365&pid=123025 |
| 9 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=210168&pid=122962 |
| 10 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121033 |
| 11 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121032 |
| 12 | £956 - £957/month per room7 Symphony Park, Manchester M1 7GB, UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121034 |
| 13 | £980/month per room7 Symphony Park, Manchester M1 7GB, UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121030 |
| 14 | £678 - £679/month per roomSalford M5 4YW, UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120131 |
| 15 | £708 - £709/month per roomSalford M5 4YW, UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120127 |
| 16 | £725/month per roomSalford M5 4YW, UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120128 |
| 17 | £775/month per roomMiddlewood Street, Salford, M5 4YW, UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120129 |
| ROW-ID | RAW-LISTING-INFO | RENT (£) | LOCATION | URL |
| ------ | ------------------------------------------------------------------- | -------- | -------------------------------------------- | ----------------------------------------------------------------------- |
| 0 | £690/month per roomChapel Street, Salford M3 5DZ, UK | 690 | Chapel Street Salford M3 5DZ UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113377 |
| 1 | £740/month per roomChapel Street, Salford M3 5DZ, UK | 740 | Chapel Street Salford M3 5DZ UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id113378 |
| 2 | £841 - £842/month per roomMiddlewood Street, Salford, M5 4YW, UK | 841 | Middlewood Street Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120130 |
| 3 | £746 - £750/month per roomSalford, M5 4ZF, UK | 746 | Salford M5 4ZF UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id122936 |
| 4 | £200/month 100, 100 Lloyd Mansions, Salford M6 6HA, UK | 200 | NIL | https://www.idealflatmate.co.uk/spare-room/salford/property-id122970 |
| 5 | £488/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | 488 | Joshua Grange Pluto Cl Salford M6 6HF UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id122929 |
| 6 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | 580 | Joshua Grange Pluto Cl Salford M6 6HF UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id123025 |
| 7 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK | 480 | Greater Manchester Manchester M31 4HZ 296 UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id122962 |
| 8 | £580/month per roomJoshua Grange, Pluto Cl, Salford M6 6HF, UK | 580 | Joshua Grange Pluto Cl Salford M6 6HF UK | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=242365&pid=123025 |
| 9 | £480/month per roomGreater Manchester, Manchester, M31 4HZ, 296, UK | 480 | Greater Manchester Manchester M31 4HZ 296 UK | https://www.idealflatmate.co.uk/auth/signup?f=b&uid=210168&pid=122962 |
| 10 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK | 850 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121033 |
| 11 | £850/month per room7 Symphony Park, Manchester M1 7GB, UK | 850 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121032 |
| 12 | £956 - £957/month per room7 Symphony Park, Manchester M1 7GB, UK | 956 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121034 |
| 13 | £980/month per room7 Symphony Park, Manchester M1 7GB, UK | 980 | 7 Symphony Park Manchester M1 7GB UK | https://www.idealflatmate.co.uk/spare-room/manchester/property-id121030 |
| 14 | £678 - £679/month per roomSalford M5 4YW, UK | 678 | Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120131 |
| 15 | £708 - £709/month per roomSalford M5 4YW, UK | 708 | Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120127 |
| 16 | £725/month per roomSalford M5 4YW, UK | 725 | Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120128 |
| 17 | £775/month per roomMiddlewood Street, Salford, M5 4YW, UK | 775 | Middlewood Street Salford M5 4YW UK | https://www.idealflatmate.co.uk/spare-room/salford/property-id120129 |
As you can see, all the entries are listed =/month=. So, I've not bothered to
create a /frequency/ column. I'm just going to move forward with that as the
default.
*Note:* There are a could of entries in the =URL= column which have valid URL's but
point to the incorrect page on the website. I'm just going to leave them in and
manually go to their listings page on the website. There isn't enough incorrect
URL's to fix this in code. On top of that, I'm more interested in the average
rent price per month than the actual locations right now.

Write Preview
Loading…
Cancel
Save