The most widely used Big Data methodologies in Energy

Big Data methodologies for Energy datasets

What is the course about:

What you’ll learn

  • Data Scientists.
  • Software Engineers.
  • Quants.
  • Big Data professionals.

Course Content

  • introduction –> 1 lecture • 3min.
  • A Big Data Method for Energy –> 15 lectures • 2hr 23min.
  • Methodologies for comparing Big Data –> 5 lectures • 41min.
  • Selecting fewer observations with set frequencies –> 5 lectures • 25min.

The most widely used Big Data methodologies in Energy


What is the course about:

Big data refers to data sets that are too large or complex to be dealt with – and this is true for electricity demand data particularly after the arrival of Smart Meters.

Smart Meters are electronic devices that record information such as the consumption of electricity, known also as electricity demand.

Imagine that 5 different locations (e.g. houses) have smart meters and every minute these devices measure the consumption of the household. Then, the amount of data produced would be too large to really perform analysis on.

When the data is too large usually a smaller dataset is selected.  And the question is then how large this dataset should be? If for example we have 1 million data points (measurements) then how much shall we select to perform some analysis? What is a “representative” dataset that we can select out of the entire dataset?

In other words, we want to minimize the loss of information. We do not want to randomly select a dataset, but rather we want to select in such a way that the information loss is minimized.

This course presents a representative Big Data methodology for application to electricity systems. This methodology can be transferred to other quantities (e.g. electricity generation) or other energy vectors (e.g. natural gas).


I am a research fellow and I lead industry projects related to energy investments using mathematical optimisation and data science. Specialized in the Data Science aspect of the Green Energy transition, focused on algorithmic design and optimisation methods, using economic principles.

  • Doctor of Philosophy (PhD) in Analytics & Mathematical Optimization applied to Energy Investments, from Imperial College London , and Master of Engineering (M. Eng.) degree in Power System Analysis (Electricity) and Economics .

Special Acknowledgements:

To Himalaya Bir Shrestha, senior energy system analyst, who has been contributing to the development of Python scripts for this course and who regularly posts on medium.



  • No pre-requisites and no experience required.
  • Every detail is explained, so that you won’t have to search online, or guess. In the end you will feel confident in your knowledge and skills.
  • We start from scratch, so that you do not need to have done any preparatory work in advance at all.  Just follow what is shown on screen, because we go slowly and understand everything in detail.
Get Tutorial