As part of a building monitoring project we used to receive a CSV with over 200,000 data point values every week. This was provided by a third party building developer, produced from a wide range of sensors within an experimental building development. The data points would include house temperature, humidity, energy use etc. My role was to make this data accessible for our partners, so I decided to use Google BigQuery.

I created a secure FTP area (accessed using a key file) where the CSV would be delivered and automated the reading of the file and writing to BigQuery. I used Python as the base code. I provided our partners with Google IAM credentials with restricted access rights to be able to read from BigQuery.

I chose BigQuery as it would allow our partners, as well as myself, to make SQL-like queries on huge amounts of data. We had over a billion data points after the 2 year project had completed. I made dashboards using React and ChartJS as well as Grafana allow us to analyse the data.