:::: MENU ::::

ESA SEOM-S2-4Sci Land and Water Multitemporal Analysis (MTA)

SEOM program

The “S2-4Sci Land and Water – Multitemporal Analysis” initiative was launched in the context of the “Scientific Exploitation of Operational Missions” (SEOM) program. The program aims at favouring: i) research and development studies, ii) development of toolboxes, iii) interaction between scientists and users; iv) training of new generation scientists in the field of Earth Observation (EO); and v) the outcomes of research in terms of data and results. These needs emerge because a very large number of images are now available to either the scientific and user communities from past/current EO missions (e.g., ERS, Landsat) and even more will be available due to the upcoming EO missions. Among those the Copernicus Sentinel program will contribute with an exponential growth of data of a large variety. This opens to an unpredictable wide range of possibilities. Among the various Sentinels, the “S2-4Sci Land and Water – Multitemporal Analysis” project focuses on the Sentinel-2 (S-2) family to perform advanced multitemporal analysis.

Multitemporal Analysis (MTA) Project

The intrinsic multitemporal nature of satellite Remote Sensing (RS) images is a high valuable property that can be used to perform accurate monitoring of the Earth’s surface and atmosphere. However, handling the time variable together with the spectral and spatial one is a complex task. For this reason, the development of advanced methods for the analysis of multitemporal data is one of the most important and complex task of the RS community.

Having 13 spectral bands, 10 to 60 m spatial resolution and a revisit time of 5 days upon complete constellation, S-2 guarantees an enhanced continuity to SPOT and Landsat missions. Due to the high temporal resolution, S-2 generates dense time series of images with a worldwide coverage. This increases the potential of analyzing dynamic phenomena and extracting multitemporal information, as well as to have a frequent coverage also in areas that maybe affected by cloud cover. With respect to previous similar missions (e.g., Spot and Landsat), S-2 provides optical images with higher spatial, temporal and spectral resolutions. In the domain of multitemporal information extraction this represents a great improvement with respect to the past, that opens to a wide range of new possibilities in the field of multitemporal analysis.

In this context, the MultiTemporal Analysis (MTA)  project is designed  to define techniques and algorithms tailored on the specific properties of S-2. The project aims at developing emerging methods for advancing the technical knowledge and capacity to analyse S-2 data and provides a series of tools to better exploit S-2 temporal information in the future.

For more information about S-2 mission and SEOM program please visit:

Application Areas

In the context of ESA S2-4Sci Land and Water  SEOM study, the MTA project investigates three main application and methodological areas:

  • Time series analysis: the potential offered by the S-2 series, in terms of dense time series of data, opens the door to the development of new and advanced techniques for the joint analyses and classification of time series of data. The methods should aim at identifying and classifying temporal patterns and spatial/temporal land cover transitions in the S-2 multitemporal dataset.
  • Change detection and attribution: due to the increased revisit time of S-2 satellite, it is necessary to develop land cover change detection strategy suitable for S-2 data that may allow a systematic identification and attribution of changes in land cover at large scale.
  • Land cover maps updating: the regular generation of land cover classification maps (specially at global scale) is a complex and time consuming problem that if done independently (different maps computed independently in different times) may affect the temporal consistency of the land cover information and even the impossibility to compare them in time. The regular update of a reference map based on multitemporal analysis may overcome some of the methodological problems to be faced ad may enhance the temporal consistency of the obtained results.

According to the project requirements, the final methods should be automatic to require small interaction with the user. Moreover, the final implementation should be scalable to work easily on large images.

Project Planning

The project activities are split into three main task:

  • Methodology Development: is devoted to a comprehensive and detailed analysis of the state of the art and of the relevant projects. The test areas and the preliminary methodologies to be developed are investigated in this task.
  • Algorithm Implementation: is devoted to the implementation and testing of the algorithms identified by Task 1 and the validation of the resulting output products. To this end a cyclic workflow approach that includes a continual development-validation-feedback loop is considered in the implementation phase.
  • Result Analysis: documents and provides a critical analysis of the feedback from scientists and institutions collected through consultations in ESA and third party workshops, symposia, and conferences. On the basis of the feedback a Scientific Roadmap document will be developed that identifies the scientific priority areas and provides the guidelines for future scientific data exploitation projects.

Project Objectives

The overall science objectives of the MTA project are:

  • Time series analysis: to perform precise agricultural analysis to fully exploit the specific properties of S-2 images. Precision agriculture generally involves better management of farm inputs such as fertilizers, herbicides or seeds. Typically, large farm fields under conventional management receive uniform applications of fertilizers, irrigation or seeds. In the framework of the precision agriculture, these fields can be divided into zones receiving customized inputs based on varying soil types, landscape position, and management history.
  • Change detection and attribution: to perform binary detection of forest changes (disturbances) mainly related to deforestation (clear cuts and logging) in bi-temporal S-2 image pairs. The deforestation problem is gaining more and more attention because of its dramatic extension and many associations are working for providing specific information for monitoring purposes.
  • Land cover maps updating: to use S-2 images to update existing thematic land-cover maps without ground reference data (unsupervised case). In the considered application, the image used to generate the existing map is considered unknown. Due to the recent S-2 launch, few thematic maps obtained with these data are currently available. Thus, the team is also investigating the possibility of updating existing maps generated with different RS data.

Theme 1: Time series analysis

Problem Formulation:

Dense time sampling increases the sensitivity to dynamic phenomena and provides a better description of them. With the increased revisit period of the complete S-2 constellation (return frequency up to 5 days), it is possible to perform accurate seasonal trend analysis. In this framework, specific applications regarding low cost precision agriculture is of great interest given that S-2 has specific bands in the Red-Edge spectral range dedicated to the study of vegetation that were not available in previous multispectral sensors (e.g., Landsat, SPOT). In the time series methodological area, the focus is set on the monitoring of irrigation of agricultural areas.

Project Output:

Satellite Image Time Series (SITS), such as the ones acquired by Sentinel-2, combine a large amount of information, compared to previous satellite generations, since a better trade-off in terms of spatial/spectral/temporal resolutions is guaranteed. These type of information becomes relevant in the agricultural analysis, where availability of dense SITS is required to map and analyze fast changing crop behaviors. The high spatial resolution offered by Sentinel-2 allows to separate and analyze single crop fields, even when their size is relatively small (lower than 9.4ha, as it frequently happens in Europe w.r.t. 19.6ha in US), with a high temporal resolution. In the literature, several methods exist that analyze the evolution of crop fields, by aggregating them, but none is fully able to work: i) at single field level; and ii) with irregularly sampled data. In this project, we developed an approach for the analysis of spatio-temporal evolution of crop fields in SITS that is able to deal with the spatial/spectral/temporal characteristics of S2 data. Experiments were carried out from S2-SITS acquired over Barrax, Spain, in the period July 2015 – November 2016, where a total of 76 images were available, but only 49 were used for the analysis, given the non-presence of clouds.

Sentinel-2 images distribution for the period July 2015 – November 2016 in Barrax, Spain.

To this aim, four steps are followed: i) pre-processing of the S2-SITS, ii) spatio-temporal fusion, iii) spatio-temporal evolution analysis; and iv) spatio-temporal information extraction. For the spatio-temporal fusion, the system identifies and separates all the crop fields cultivated at least once over a given area based on their vegetation spatio-temporal evolution.

RGB images

Normalized Difference Vegetation Index (NDVI)

Multitemporal crop field map

Barrax (Spain) agricultural area over the period from July 6th 2015 to November 20th 2016.

Example of the original and continuously reconstructed mean NDVI for a single crop field.

For the spatio-temporal evolution, the system reconstructs continuous and regular sampled S2-SITS at single crop field level by means of an adaptive non-parametric regression model. For the spatio-temporal information extraction, two types of information are considered: i) extraction of phenological parameters and ii) creation of cumulative indices maps. For the phenological parameters extraction, the trend of the mean NDVI, at single crop field level, is used to derive their phenological behavior and create maps related to beginning, middle and end of season. This same information is then used to create maps indicating information such winter/summer cultivation period.

Beginning of season

Middle of season

End of season

Phenology estimation maps over a 12 months period.

Phenology estimation for three sample crop fields: beginning of the season, middle of the season. end of the season.

Cumulative NDVI

Cumulative NDWI

Mean cumulative NDVI and NDWI with a value higher than 0.5.

For the creation of cumulative indices maps, well-known thresholds from the literature are used as starting point over given radiometric indices in order to generate a map offering critical or relevant information at single crop field level. Such are the cases of the NDVI and NDWI indices, where a threshold over 0.5 can provide information about how many times a field has been cultivated over a given period, and how many times there was presence of water on that field.

Theme 2: Change Detection and Attribution

Problem Formulation:

Change detection has been widely used to assess  land use/land-cover changes. Among them, one of the most interesting is the analysis of forest changes, where the introduction of new imagery from the S-2 satellite is expected to bring significant improvements with respect to the past. Indeed, changes can be now identified at time/spatial/spectral scales never considered before. In this context, for the change detection and attribution area the binary detection of forest changes (disturbances) mainly related to deforestation (clear cuts and logging) in bi-temporal S-2 image pairs is considered.

Project Output:

More than ever, forest resources exploitation is strengthened all around the globe. In this respect, the Indonesian forest has been devoted a large interest in the recent years due to an intense and increasing lodging activity associated to it. Indeed, large portions of the forest are cleared to the purpose of harvesting wood and creating land for farming (especially palm oil plantations). This process can be monitored as the typical patterns in which the cuts are made allow for a precise identification by means of remote sensing systems.

Two study areas analyzed in this project. The largest one is the Kalimantan Island (country scale analysis), the smaller one is corresponding to tile 49MFT in the MGRS system (regional scale analysis).

The scientific parameters of the Sentinel-2 mission determine a new standard for the utilization of high resolution optical imagery for the study of vegetation and associated phenomena. The detailed spectral range available at high resolution allowed us to implement an adaptive unsupervised
approach based on multivariate CVA that automatically identifies patches of clear cuts on Sentinel-2 image pairs. The proposed multitemporal system is able to process time series and aggregate the results making it possible to both: 1) recover partial loss of information due to cloud coverage, and 2) keep track of the spatial-temporal evolution of the cut patterns.

Time series of Sentinel-2 images over tile 49MFT acquired between December 2015 and April 2017 with cloud coverage less than 50%.

The system takes as input a time series of Sentinel-2 images acquired over the same area, it triggers a separate process for all the possible bitemporal pairs and finally it aggregates the results by returning a “heat” map of the detected changes where the “temperature” is associated to date of detection. At the core of the method there is the capability of estimating the Gaussian parameters of the change class (a transition between vegetation and a specific kind of soil related to the cut) in the bi-temporal stacked domain and to extract all the pixels that belong to its neighborhood.

Theme 3: Land Cover Maps Updating

Problem Formulation:

Many techniques have been developed to update maps without ground reference data (unsupervised case). However, all these methods assume to have reference data or a thematic map for at least one image of the time series. Even though this is reasonable in many real applications, it is no longer affordable when considering the need of updating land-cover maps at national or global scale. For this reason, in the last methodological area the time series of S-2 images is employed in order to update an existing thematic land-cover map generated with different RS data by assuming that: i) no reference data are available; ii) the origin of the thematic map to be updated is unknown.

Project Output:

Combining existing thematic vector products and recently acquired satellite images to generate regular updated maps is extremely interesting at operational level. However, employing these maps is not straightforward. Besides the problem of having outdated and possible misclassified maps, dealing with vector land cover maps is an ill-posed problem that has to be accurately modeled. Experiments have been carried out in Czech Republic, by updating a Crop Type Map representing the main cultivations of 2014-2015 in a completely automatic and unsupervised way.

Land Cover Crop Type Map (t1) representing the main cultivations of the Czech Republic for 2014 – 2015.

In this research, we developed a framework that aims to understand and model the LC vector map domain to transfer knowledge for the training of classifiers on recent multispectral data. The first component of the proposed framework performs a spatial decomposition of the considered thematic product, to increase the possibility of identifying pure pixels associated to valid labels (i.e., belonging to the predominant semantic class of the polygon).

Besides the spatial and pixel decomposition, the map legends usually present semantic-classes (e.g., the “crop” label is assigned to all pixels representing different cultivations, i.e., natural classes). This leads to a semantic gap between the map and the set of natural classes that can be discriminated by the spectral information provided by the remote sensing images. To extract an informative and representative set of labeled samples from the thematic map, the second component of the proposed framework performs a semantic decomposition of the legend map.

Finally, the set of informative labeled samples is used to transfer knowledge from the initial outdated vector map on updated map by classifying a recent time series of Sentinel-2 images.

Project Team

The partnership of the MTA project supported by ESA is led by the Remote Sensing Laboratory (RSLab) at the Department of Information Engineering and Computer Science of the University of Trento (UoT) and involves the Remote Sensing for Digital Earth Unit at the Fondazione Bruno Kessler (FBK) research center.

The RSLab coordinated by Prof. Lorenzo Bruzzone has a huge know-how on the definition, implementation and application of techniques and algorithms for the analysis of multitemporal images. RSLab has a widely recognized leadership in multitemporal data analysis and processing of time series in terms of change detection, and analysis of long time series and updating of land-cover maps. This is demonstrated by the quantity of RSLab publications on these specific topics (and their quality proven by citations), by the projects/applications managed by the group, and by the many initiative related to the temporal variable in remote sensing RSLab has managed. RSLab is the Prime Contractor of this proposal and has the scientific responsibility of the analysis of the state-of-the art, of selecting/defining the most effective methods and algorithms to be implemented in order to satisfy the requirements of the call, and of defining the road map for future activities. RSLab will then contribute to the implementation and to the validation of the algorithms.

The RSDE is an application oriented research group developed within the internationally well-known FBK research center. This Unit is built around the large know of the leader (Dr. Francesca Bovolo) in RS, who is a top expert in the field of multitemporal data analysis. RSDE in the project will lead the implementation, testing and validation of the developed tools on S-2 multitemporal datasets taking into account the large expertise and computation infrastructure the center has on the processing of large quantity of data. Moreover RSDE will contribute to the selection and implementation of the algorithms and to the definition of the road map.