DEVELOPMENT OF METHODOLOGY FOR PLANT PHENOLOGY MONITORING BY GROUND-BASED OBSERVATION USING DIGITAL CAMERA

When monitoring phenology at ground level, it would be more important to continue observations in long terms and to detect the timing of various phenological events such as leafing, flowering and autumn senescence. In this study, to develop the methodology for plant phenology monitoring by using digital camera, we examined how multiple image indices, which are derived from multi-temporal visible images, respond to the changes of colors of leaves and flowers for several target species of plants, and tried to detect various phenology events by tracing time series changes of the coordinate in the feature spaces of two indices. As a result, we found out that it was possible to understand the characteristics of the phenological events for different species from each image index. Also, it was identified that the utility of combination with two indices would be effective to detect the timing of phenology events in the feature space of two indices. In the actual phenology monitoring, it would be effective to use a single index for understanding the seasonal characteristics and to use the combination of two indices for detection of the timing of phenology events by tracing the time series changes in the feature space.


INTRODUCTION
The timings of leaf appearance, autumn senescence, and flowering are different and shifting year by year.These phenomena are occurring because of abnormal weather in short term and climate variability in long term due to global warming (Ono et al., 2015).The impacts of abnormal weather and climate variability on vegetation cause various problems.Examples for these are the difference of timing of events across species such as food chains and pollination (Kudo et al., 2004;Doi et al., 2008), the change of leaf appearance and autumn senescence that leads to change of turbulent exchange of water and energy (Moore et al., 1996;Sakai et al., 1997, Richardson et al., 2007), and the lengthening of growing season that leads to change of primary productivity, photosynthesis, carbon balance, and CO2 concentration (Ide and Oguma, 2010).To understand such impacts on vegetation, it is important to observe changes of leaves and flowers, so called phenology, day-by-day and continuously.Phenology observations are being conducted in many countries and regions (Morris et at., 2013).Observations of plants should be conducted in individual species and plant scale, and should be done every day, because timings of changes differ among individual plants and species, and change of color occurs in short term.Therefore, ground-based observation would be suitable when observing phenology.Traditional field observations have been conducted for over a century by individuals or groups of people.For example, Japan Meteorological Agency has been observing trees for many years to know the overall aspect of weather.They observe index trees of more than 10 species at 102 places.Field observations can be done in detail, but it is labor intensive, subjective, and can only be done in places where people can access [Sonnentag et al., 2012].That is, to understand various phenological events and to archive these records, it is necessary to monitor the same plants from fixed points continuously for long-terms.For long-term phenology monitoring, continuous observation is effective by taking images using digital camera at the same times day-by-day at a fixed point.The images taken by digital camera basically consist of RGB channels that correspond to the visible region.However, to understand vegetation changes, characteristics of near-infrared region are generally used, because the vegetation activities reflect in not only visible ray but also near-infrared.On one hand, the ground-based phenology observations using digital camera have been carried out in worldwide recently, for example PEN and PhenoCom (Nagai et al., 2016;Brown et al., 2017).As for phenology monitoring on the ground level, it would be more important to continue longterm observations and to detect the timing of various phenological events, than to understand the details of vegetation activities.
From the mentioned background, the purpose of this study is to develop the methodology for plant phenology monitoring by ground-based observation using digital camera.The most important key issue is to detect the phenological events such as leafing, flowering and autumn senescence efficiently by using multi-temporal visible images.We recognize this phenological event visually at change of seasons in Japan.In this study, we adopt to utilize multiple image indices proposed by previous studies and examine how these indices respond to the changes of colors of leaves and flowers for each target plant.Because image indices are known as one of useful methods for image enhancement and feature extraction.Furthermore, we propose the method for tracing time series changes on the coordinates in feature space of two indices to detect different timings at various phenological events for each target plant.

Study site
The site is located at Tokyo University of Agriculture and Technology in Tokyo, Japan (35°39.94'N139°28.37'E).The annual mean temperature is 15.0°C with an average annual precipitation of 1529.7 mm.There are typical four seasons of spring in March -May, summer in June -August, autumn in September -November, and winter in December -February.The plants in the site are mostly deciduous broad-leaved trees and evergreen broad-leaved trees with exceptions of a few coniferous trees.As for deciduous broad-leaved trees in the site, generally the leafing finishes by June and the autumn senescence starts in October.Thus, this site could be decided as adequate study target.

Observation
To observe the color of the leaves and flowers on trees, we used a single-lens reflex camera.The camera was installed on the roof of a four-story building with the angle of depression at 30° and the center direction of the field of view (FOV) at 70° from true north.We horizontally put a grayscale board, which has about 44% reflectance, in FOV of camera to normalize the color and the brightness on images effected by different sunlight conditions.The camera has been taking images every 10 minutes during daytime from April in 2017.The used camera and settings are shown in Table 1. Figure 1 shows the view of our used camera and the grayscale.The maximum input value (xmax_) is defined as the average of R, G, and B digital numbers in the area of the grayscale board (Figure 1).In this case, we set each maximum output value of R, G, and B (ymax_) as ymax_R :214.8,ymax_G :226.6, and ymax_B :231.5 based on the laboratory measurement of spectral reflectance.As for the minimum input value (xmin_) and output value (ymin_), we set them as 0. Thus, eq.1 is simplified as eq.1'.
(1') By using eq.1', the input values (x) of digital number in image are converted to the output values (y).Each of all obtained images is converted to the normalized image.
All normalized images are used to calculate image indices.
According to these indices, it seems that we can use 2G_RBi, 2rG_rRBi, and ∆NGB to describe the greenness of the color of leaves, while VIgreen is used to describe autumn senescence.
The equations and explanations are shown as follows.
2G_RBi: This index shown in eq.2 expresses the greenness of objects and is also called as Excess Green Index.2G_RBi is calculated by the sum of the difference in green and red and the difference in green and blue channels, so that the value of index is shown in digital number.It is widely used in phenological observations.(5)

RESULTS AND DISCUSSION
We used images taken from April 29 to December 31.As for precipitation in this season, there was not much rain during the rainy season in June, but had much rain in August.
The main trees that can be seen in the images are shown in Figure 2.There are 8 species shown in Table 2.In this paper, we focus on 4 deciduous trees shown by red line in Figure 2, and discuss the results of our analysis.These 4 trees generally have the following seasonal changes; Z. serrata has green leaves that turn yellow, then red in autumn senescence.
G. biloba has green leaves that turn yellow in autumn senescence.
A. x carnea has green leaves that turn yellow, then red in autumn senescence and red flowers in spring, B. florida has yellow green leaves that turn red in autumn senescence and white flowers in spring.

Effect of weather and sunlight
To investigate the effect of weather and sunlight on values of the image indices, we compared the differences of the values of each index between clear sky, sunny, and cloudy days.We chose dates that were not in flowering, leaf appearance, nor autumn senescence season, and that were close to each other.We selected the dates of July 21 for clear sky, July 27 for cloudy, and July 28 for sunny.To reduce the effect of bi-directional reflection, we examined the diurnal differences of each index on clear sky day shown in Figure 3, for example.The indices are shown as stable relatively from around 10:30 to 12:30, so that we chose the images that were taken within 1 hour before and after the time when the direction of the sun was 90° from the direction of the camera.Thus, we used 13 images that were taken when the sun was around the direction of 160°, because the center direction of FOV is 70°.These selected images were taken around 10:10~12:10 in spring, 10:30~12:30 in summer, 9:50~11:50 in autumn and 9:30~11:30 in winter.

Seasonal characteristics of image indices
To examine how the image indices respond to the changes of leaves and flowers of trees, we investigated the temporal patterns of the daily average of each image index during this observational period.There were some days in which the camera could not take images, so we only used days which had at least 9 images available in total 13 images.Figure 5 shows the seasonal fluctuation of 4 image indices for 4 trees.Here, we discuss about the characteristics of each tree shown by each image index as follows.
Figure 5.The seasonal fluctuation of each image index for 4 trees during from 29 th April to 31 st December.Since there was no big change from June to August, it is omitted in this figure .Z. serrata: There is one peak in all four image indices in the middle of May.When the values of the image indices are increasing, the tree is leafing.In the images, the canopy which started as brown, which is the color of the branches, becomes green, which is the color of the leaves.When the values of the image indices gradually decrease after the peaks, the color of the leaves become darker.After September, VIgreen and ∆NGB are slightly decreasing until the end of October, then all indices drastically degrease, because of starting the tinted autumnal leaves.After that, the leaves have been falling until they fell off completely at the end of November.During the deciduous season, all indices show the quite small increase.

G. biloba:
There is an increase or decrease in the values of all four image indices in May and early August, and a peak in late July for 2G_RBi and 2rG_rRBi.In May, the values of 2G_RBi and 2rG_rRBi decrease while VIgreen and ∆NGB increase.In the images, G. biloba already has leaves when the observation starts on April 29, and the color of the leaves become darker green in May.The values of VIgreen and ∆NGB responded to the color of the leaves becoming less "reddish" and "blueish", and the values of 2G_RBi and 2rG_rRBi responded to the color of the leaves becoming darker.There is a peak in the values of 2G_RBi and 2rG_rRBi in late July and a drop in all four image indices in early August, but we could not find the reason for this on the images.It might be because of the long spell of cloudy weather.After September, all indices decrease by the end of November.Especially VIgreen is obvious, because the color of leaves in this tree changes yellow in autumn senescence.After that, three indices except for VIgreen decrease greatly.In contrast, VIgreen is close to 0 from the negative value.

A. x carnea:
There are a rapid decrease and increase in all four image indices in the middle of May, a gradual decrease after they increase the peak in late May except for ∆NGB which has a peak on July 4, and a period with smaller values in early August like the one in G. biloba.A. x carnea starts with leaves on and no flowers, and when the values of A. x carnea in all four image indices decrease, the red flowers bloom, and when they increase, the flowers drop.After that, the color of the leaves becomes darker, which is when the values of image indices decrease.The reason for the peak of ∆NGB could not be found.We could not find the reason for the period with smaller values in early August either.During autumn, 2G_RBi and 2rG_rRBi slightly increase, and then decrease from the end of November.

B. florida:
There is a peak in the first half of May.After that, the values of 2G_RBi and 2rG_rRBi gradually decrease while VIgreen and ∆NGB is decreasing and increasing.There is also a decrease and increase in late August in all four of the image indices.B. florida already has white flowers and yellow green leaves at the start of the observation, and the flowers drop in the first half of May, which is when the values of image indices are increasing.The difference of values between the start and the peak is small in ∆NGB compared to the other image indices because ∆NGB is calculated by G and B, and the digital numbers of R and G do not change much.The digital numbers of B change in the period, but the digital number of B is small, so it did not affect the value of ∆NGB much.After the middle of May, the color of the leaves become darker which was when the values of 2G_RBi and 2rG_rRBi gradually decrease.The values of VIgreen and ∆NGB increase and decrease in this period, but we could not find the reason for this.We also could not find the reason for the decrease and increase in late August.After September, all indices decrease until the end of November, and then the indices increase again.This was affected by the object behind tree after the leaves fell off.We compared the fluctuation of the image indices between trees.As looking at 2G_RBi, B.florida has the largest value overall and A. x carnea is the smallest, however, when we look at 2rG_rRBi, its difference is smaller.This was because 2G_RBi was directly affected by the lightness of the object, but in contrast, 2rG_rRBi was less affected because it used normalized values.As for VIgreen, B. florida has the smallest value overall.This is because the color of the leaves of B. florida is the closest to yellow.As for the overall of the image indices, the variability of the value of 2rG_rRBi was larger than 2G_RBi.This was probably because of weather, so it seems that the value of 2rG_rRBi was more affected by weather than 2G_RBi in this study.Also, B.florida has the least variability.This is because it has the least shadows.Regarding to leaf phenology, 2G_RBi and VIgreen were useful at the leafing stage, ∆NGB and VIgreen at the autumn leaves stage, and either 2G_RBi, ∆NGB and VIgreen during the falling leaves stage was useful.As for flowering and falling of flowers, it is considered that VIgreen is effective for red flowers, and 2G_RBi and 2rG_rRBi are effective when the brightness difference between leaves and flowers is large.

Tracing the time series changes in feature space
Through the examination of seasonal changes of 4 indices for 4 trees, we could understand the phenological characteristics of each index for each tree.Here, to detect the timing of phenological events, we trace the time series changes on the coordinate in the feature space of two indices.As for 2G_RBi and 2rG_rRBi, both two have almost the same characteristics, so 2G_RBi was not used in this examination.In addition to 2rG_rRBi, ∆NGB and VIgreen, we also use a new index of VIGB which replaces R of VIgreen with B (eq.6) to use for comparison with VIgreen.From the above results, it is considered that the combination of image indices that is most effective in phenology monitoring is VIgreen and VIGB where the most characteristic changes were observed.
As the result of detecting the phenology events and the timing in the feature space of VIgreen vs VIGB, the time series changing of G. biloba on the coordinate in this feature space is shown in Figure 7, for example. of the end of leafing was around on 20 th May, the date of the peak of yellow leaves was on 30 th November, then the leaves were falling and finished in the end of December.

Efficient utility of multiple image indices
The advantage of phenology observation using time series images is that seasonal changes can be relatively easily investigated using image indices showing colors.However, in case of using a single index, there were occasions when the phenomenon was difficult to detect depending on the species of trees, as there were cases in which the value varied due to the weather and the phenomena were not matching with the change in value.
On the other hand, the advantage of tracing the time series changes in the feature space of two indices is easy to identify the timing of phenology events, because it uses the change in coordinates by the two image indices.Therefore, in the actual phenology monitoring, it would be effective to use a single image index to understand the seasonal characteristics and to use the combination of two indices for detection of the timing of phenology events by tracing the time series changes in the feature space of two indices.

CONCLUSION
To develop the methodology for plant phenology monitoring by using multi-temporal digital camera images, we adopted to use multiple image indices and examined how these image indices respond to the phenology events such leafing, flowering and autumn senescence of trees, and also proposed the method to detect the phenology events in the feature spaces of several combinations of two indices.
As for the used digital images, in order to reduce the effects of sunlight, we normalized the color and brightness using the grayscale board on all obtained images.As a result, the influence of bi-directional reflection on image indices could be reduced by only using images taken within 1 hour before and after the time when the direction of the sun was 90° from the direction of the view of camera.Through this study, we found out that the seasonal characteristics of phenological events could be understood by using image indices.2G_RBi and 2rG_rRBi detected the change of greenness and darkness of the color, while VIgreen and ∆NGB detected the change of greenness, and especially in flowers, VIgreen was effective.Furthermore, it was possible to detect the timing of phenology more efficiently by tracing the time series changes on the coordinates of two indices, especially the combination of VIGB and VIgreen.
As for future works, we will develop the two-indices model in the feature space of two indices to detect the phenological information automatically.
number of Green channel R; digital number of Red channel B; digital number of Blue channel 2rG_rRBi: This was modified for 2G_RBi to reduce the effect of sunlight illumination under different weather conditions.It uses the ratios of RGB values instead of digital numbers (eq.3).This index is normalized by dividing the difference of green and red by the sum of both (eq.4).The index value can be shown from -1 to 1.The positive value closer to 1 shows greenness and the negative value closer to -1 shows redness.The value closer to 0 shows yellow.This index shown in eq.5 was also developed to reduce the effect of solar radiation by using the normalized green and blue values divided by the average of 3 channels, and expresses the greenness of objects.

Figure 2 .
Figure 2. ROI (region of interest) of the target trees

Figure 3 .
Figure 3.Diurnal differences of 4 indices on 21 st July (clear sky)The average with standard deviation of each image index are shown in Figure4.The value of 2rG_rRBi slightly increases under clear sky.It is pointed that 2G_RBi is influenced by sunlight condition(Ide and Oguma, 2010), so it seems that this was because the direction of sunlight became closer to camera direction, which was direct-light condition in image.However, there was not much difference between the average values under different weather conditions (Figure4).Most of the standard deviation was largest in clear sky.The reason seems that much leaves illuminated by direct sunlight and shadows are mixed under clear sky.Overall, there was not much difference of the average values under clear sky, sunny, and cloudy conditions, thus, we decided that it would not be a problem to use the selected 13 images per day as the daily averaged data under all weather conditions.
day 2rG_rRBi = (rG-rR)+(rG-rB) day day day • Z. serrata • G. biloba • A. x carnea • B. florida Figure 6.shows the results of time series changes of 4 trees on each coordinate in 4 feature spaces of VIgreen vs VIGB, VIgreen vs ∆NGB, VIgreen vs 2rG_rRBi and ∆NGB vs 2rG_rRBi.As regard to Z. serrata, in the four feature spaces, both indices increase (shifting to upper right) in the leafing stage, both indices decrease (shifting to lower left) in the autumnal leaves and were close to the origin during the falling leaves stage.For G. biloba, in the autumn leaves stage VIgreen decreased and VIGB increased in the feature space of VIgreen vs VIGB (shifting to upper left), but in other feature spaces both indices decreased.Both indices in all feature spaces were closed to the origin in the falling leaves stage.As for A. x. carnea, both indices in all feature spaces decreased during flowering period.At the time of full bloom, VIgreen and ∆NGB increased in 3 feature spaces except for the combination of VIgreen and ∆NGB, VIGB and 2rG_rRBi which decreased.Both indices were increasing in falling flowers.B. florida had little change in VIgreen and ∆NGB except for the coordinates of VIgreen vs ∆NGB in falling flowers, and VIGB and 2rG_rRBi increased, and both indices had little change in the feature space of VIgreen vs ∆NGB.In these 4 feature spaces by combinations with two indices, the changes in the color of trees and the changes on coordinates generally agreed.Especially for autumn leaves the coordinates changed to the space showing yellow or red color in any tree, and become close to the origin at the time of fallen leaves.In addition, changes in coordinates during the whole observation period were similar between Z. serrata, and B. florida, and G. biloba and A. x. carnea.

Figure 6 .
Figure 6.Time series changes of 4 trees on each coordinate in 4 feature spaces of two indices.

Figure 7 .
Figure 7. Time series changing of G. biloba on coordinate in feature space of VIgreen and VIGBThe changes of coordinates start at upper right, move down, stable there for about three months, shift to upper left i.e. the space shown yellow on this coordinate, move down again, and approach the origin.In other words, we can explain that the date

Table 1 .
Used camera and settings

Observation camera and the grayscale board in the image 2.3 Image normalization
To reduce the effects of sunlight by different time and season on image, we normalized the color and brightness on the original images using the function of liner conversion (eq.1).