Big Data is the last link in predicting weather conditions. Around the world, thousands of companies, scientific centers, institutions, etc., are using Big Data to find patterns wherever they are, the big data. In meteorology, a science that also has a huge and huge amount of data, Big Data also has its useful applications. This modern and powerful tool, it can be used in multiple ways. Despite being named as a single thing, it can achieve many different predictions depending on what you are looking for. Of course, it has also come to meteorology, and here we are going to tell you what it does and how.
First of all, let's remember that anticipating time has always been one of the primary needs of human beings. Thousands of years ago, weather forecasts were very important, even more than today, for survival. Technological development was not so cutting edge, any instability could have serious consequences. Although there was always this need to prevent the weather, it was not until the arrival of Aristotle that we can coin the term meteorology. He called it "meteorological", the name he gave his book, around 340 BC.
Big Data in forecasts
The logic of atmospheric behavior has not stopped developing since then. Each time faster. Going through the thermometer that Galileo invented in 1607, to computer simulations from data collected by satellites. Right now, we are faced with Big Data, many agree that it is the most revolutionary tool since the internet exists and is not for less. As if it were a science fiction future, today we can say that it is real.
As we have commented, Big Data begins to take charge today, of giving that other point of view to meteorologists. Where they could not go, or believed they were right without being, Big data shows you what was hidden or unnoticed, also with a level of precision never reached. There are companies that already offer these services today. Institutions, governments and companies that make use of big data to anticipate the climate. But how is this whole process? How is it done? How do we benefit? Next we will see and understand how this entire process of technological innovation is possible.
How does Big Data work?
Roughly, Big Data puts more than looking at the sky to focus on data, and that they are processed correctly. So that you can understand more in its magnitude the implication with meteorology, first it is necessary to explain how it works.
Big Data has its core of operation in what is called the 4 V's.
Volume
This means the amount of data. All this amount of data collected is what is known as the volume. It may vary depending on what is applied, sometimes we have a lot of data and other times "less". That is, we can go from 1.000 million data to several trillion, depending on which is analyzed.
Speed
That is to say, the rate at which data is generated. They come from the need to capture, store and process them. The more data captures there are, the faster they are stored, the more there is to analyze. Speed is of double importance in weather forecasts, since events occur in real time, and must be processed as soon as possible.
Variety
Sometimes there is a format of how that data comes, other times others. Each type of data has its own classification. Other times some are missing (there are techniques to fix this, or the errors would be huge) and other times they come in video forms even. There is a very different mass of data, which in Big Data is responsible for putting an order, a logic to be well analyzed. For example, temperature measurements from a thermometer "cannot" be put in the same package as satellite measurements from a front.
Veracity
Related to the parenthesis of the previous point. It means that the data finally comes clean, without "weird" things. Big Data management teams must have an impartial team trained to maintain a good structure. The consequences of a bad veracity of the data have very negative effects. To get an idea, it would be as if a group of mechanics finished the repair of a car, and they forgot to screw two wheels.
Example on the veracity of the data
We have many records from many areas. Let's imagine we have temperatures, humidity levels, winds, etc. But, we have a failure, and we are missing some temperature records for some area, for whatever reason, and we cannot access to know what temperature has been recorded. We have a total of 30 data, and two of them, without temperature finally.
What could be done, for example, is to calculate the average temperature of those regions to determine precisely the possible temperature that can be counted on in the missing record, but also with very small margins of error. Values are spare parts, and then the calculation can be put into practice. Had this data been missing, the computers would not have recognized it, creating a black hole in the data, and totally wrong predictions.
How is it done?
In meteorology, as in any field, the data comes in the form of variables. That is, each one is processed in the way it belongs. And although it seems very convoluted and complicated, the task becomes "easy" for Big Data analysts. The variables that we can record in meteorology, although they are still data, they may belong to different families. That is, a variable is any data that can be classified, but they are not always the same.
The image above, provided by NASA, shows the example of currents around the planet. In the case of NASA, they have a large number of satellites that allow them to observe and measure phenomena around the globe in real time.
Big Data can read every trace that something leaves about something, and that can be considered data. Many when thinking about Big Data, they will quickly think about when we use mobile phones, surf the internet, click on a page, buy an item online, or “like” it on Facebook. That is only a "small" but dense part, yes, it is very reliable and well coded. But in turn, we leave a physical / virtual trail, like the GPS location of where we are, thanks to mobile phones. Here we already begin to mix the virtual world with the physical one. And of course, physical movements, physical purchases, according to age, what we choose, all this is always archived, and of course, it can translate into more and more data.
Variables can be categorical
Categorical variables are those that represent limited values or variables that do not necessarily mean a specific magnitude. They represent the quality of something they describe. Basically their particularity is the limitation of what they represent. They can be classified into two fields.
Nominal categorical variables
They are those that represent things in the same field without a logical connection each. For example: The name of the regions that indicate where the records are from, such as the city, autonomous community, a postal code, etc.
Ordinal categorical variables
They are those that can represent the magnitude of something, such as the Douglas scale in the wave level, the level of the scale with which tornadoes can be classified according to their magnitude, etc.
Variables can be numeric
Numerical variables are those that represent values or variables within a magnitude and can be measurable. They represent quantitative values. Their particularity is that they can represent a very large range of measurements in meteorological phenomena. They are classified in two ways
Continuous numerical variables
Continuous variables are those that are in charge of measuring something established. Examples of them would be the humidity index, temperature, wind speed, amount of rain, etc.
Discrete numerical variables
These are the ones they keep track of something established. That is, the number of times it has rained in a year in a region, the number of times it has snowed, etc.
All variables are processed
Once all the variables have been classified, they are processed thanks to computers, always supervised by analysts of Big Data. Until a few years ago, the amount of data that was available, despite being a very large number, there were no problems to be analyzed by data analysts. Big Data analysis, however, is responsible for the analysis of this massive data, where the analysis processes that have been common until today, would take a long time (we even talk about days) to give an answer. Not only that, Big Data is more efficient and accurate, by "playing" with the variables between them.
All this originates what we have previously commented on the 4 V's of Big Data, achieving speed, reliability and weather models that give incredibly accurate forecasts in a super short period of time.
Big Data as a nascent discipline
A good example would be to talk about the ACCIONA company, which has a Renewable Energy Control Center (CECOER). It is the largest center in the world where the objective is to provide solutions in real time, of the millions of data that are collected from its facilities, both biomass, wind and solar energy. It produces about 3000 annual schedules that take all this data to adjust to the required demand. Another advantage of CECOER is the reception of incidents that they have from their facilities, thus 50% of them are solved remotely. The remaining 50% are physically fixed by the operators. In this way, Acciona gets its renewable energy, more than being an alternative energy, is today a solution.
CECOER ACTION
Another important fact about Big Data today is the shortage of data scientists. It's a nascent field, and that has run up against certain preconceived standards. Can Big Data really help so much in the evolution of forecasts, report benefits to companies, be able to anticipate so many things and justify the cost of big data analysis? Yes. But it is something that has been seen little by little. The growing demand for data scientists has paralleled the results and by understanding the need for them in all places. It is true that there are already many Big Data teams working, with spectacular results, but it is right now where we find that there is a greater demand. Big Data analysts are being highly sought after.
In consecuense, we are living the revolution that they imply in development, but from the beginning. Like any industry, we are now witnessing its potential, but it has not been fully developed, this is something that time has in store for us. One thing is already evident, its current potential, the other, how far it can go. Your results will not leave us indifferent.
IBM model map
IBM's The Weather Company is a private company that offers up to 26 million daily forecasts about the weather. IBM from the beginning has stood out, also along with Google, for being one of the most pioneering companies in the field. The Weather Company is extremely committed to helping people make informed decisions about the weather. It is the largest network in the world of personal weather stations. The world's largest brands in aviation, energy, insurance, media and government depend on The Weather Company for data, technology platforms and services.
Big Data against Climate Change
United Nations Global Press, a big data initiative of the United Nations and Western Digital Corporation, have signed an alliance to fight together against climate change. This project led by the UN and Western Digital Corp., bring together digital innovation scientists from around the world to attack the problem in a more efficient way. Among them, we find collaborators from very different sectors among them. BBVA, Orange, Planet, Plume Labs, Nielsen, Schneider Electric, Waze ... are some of those who participate in this project.
We also find the Barcelona Supercomputing Center (BSC), It is the 4th model in the MareNostrum series. A supercomputer for Big Data analysis key in many fields, among them is also the fight for Climate Change. It was put into operation at the end of June of this 2017. It is the third fastest computer in Europe, an investment has been made in it for its installation of 34 million euros by the Ministry of Economy, Industry and Competitiveness of Spain. It has a capacity of 14 Petabytes, that is, 14 million Gigabytes. It reaches 11,1 Petaflops, that is, the barbarity of 11.100 billion operations per second.
Big Data in the future of meteorology and in our lives
In a changing world, where changes are becoming faster, and increasingly surprising, it is difficult to predict the future of something. What we know for sure is that Big Data has come to stay, and that the forecasts made both in the weather and in other areas leave us perplexed. Some will remain skeptical, others will deny it, others will see it as something far off. But the truth is, we are already living with it.
To this day we know that Big Data anticipates many rains, hurricane seasons, and even with great precision the number of medals that a country can win in the Olympic Games. It also anticipates who, where and when a crime is going to be committed (if anyone has seen the "Minority Report" movie it has crossed their minds, right?). Big data is rapidly moving towards anticipating the future of many areas, and it is that even Amazon begins to anticipate it, and recently it has begun to make shipments even before customers make purchases. The future was until today, often uncertain. But it's changing the future is predictable.
We know that its potential will grow. Who knows, it may be rash to anticipate who anticipates (Big Data) something. But with enough data, Will Big Data be able to anticipate the global climate with enormous anticipation? Yes. Just as you can anticipate that our actions would give different scenarios to those previously given, because any action has its echo in the future, and Big Data knows it and re-evaluates it, giving another new scenario.
Everything can be anticipated. Will we be able to know in the near future what will happen to us? What problems will we face? When and where will a hurricane strike? What are we going to have to continue to solve it? As techniques improve, computers improve in efficiency and speed, this field continues to develop ... Most likely is that rather than answering "who knows", perhaps the most appropriate thing will be to say "let's ask Big Data."
BA Partners | Willis Update | POT