LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models

Thank you for your valuable and insightful reviews.

In our ablation studies, we have explored some simplified versions of LITE, e.g., replacing the frozen LLM with a linear layer (LITE-LLM), and removing the multi-granularity information (LITE-mtg). However, such changes will compromise some performance degradation. From the experimental results, we observe the most necessary parts aside from language including vision (temporal trend image), multi-granularity information to handle distribution shifts, and imputation to handle incomplete observations. The other designs, e.g., the SMoE layer for imputation, can be removed or replaced under some circumstances. Our results in ablation studies can provide some useful guide for future research or applications.

Thank you for your valuable reviews and we have conducted experiments according to your suggetions.

Latest baseline. While the modeling of environmental ecosystems is crucial to people’s livelihood, this area is under-explored. We summarize the past efforts at the accurate modeling of environmental ST data and propose a novel multimodal LLM-based method that can serve as a new strong baseline and provide useful insight for future research.

Replace LLM decoder with LM. We have conducted experiments using a smaller LM - distilbert, to replace the frozen LLM (LITE-LM).

Although we tune the LM, it performs worse than the LLM version. This is due to its limited ability to fuse the multimodal representations under domain instructions. In contrast, the LLM in LITE is frozen and performs much better.

Week and year data. In LITE-year, we keep only the week data. In LITE-week, we keep only the year data. We also include results of LITE-mtg (remove week and year data) and LITE for comparison.

Our observations are: (1) both week and year data contribute to the superior performance of LITE; (2) week data is of greater significance than year data in terms of the above benchmarks. (3) combing week and year data, which is our multi-granularity information, yields best performance.

Linearized data to natural language? Linearized data lacks the smoothness of natural language, which is important for modeling the inter-variable correlations in the semantic space.

How to find missing variables? For any missing variable, like air temperature, we represent it as “air temperature: None” in the linearized data. After transforming it into natural language, we replace “None” with a special token, [MASK], for further imputation.

The role of imputation loss is to guide the imputation of missing variables. When we remove it in LITE-imp, this lead to a performance drop.

Thank you for your valuable reviews and for acknowledging the novelty and effectiveness of our approach.

Thank you for your valuable and detailed reviews.

Variables in the CRW-Temp dataset includes rainfall, air temperature, etc. For instance, the rainfall on a given day can influence the air temperature.

Eqn 3, 5, 9. Example of Eqn 3: time: 2017-01-07, radiation: 0.5, air temperature: 25, precipitation: 1.2, spRH: 0. For Eqn 5, please refer to the fourth to second lines before it. In Eqn 9, we encode the temporal trend image with a swin transformer.

$\mathbf{x_{r,t}}$ is a data point, which has K features.

Notations about ‘missing variables’. Please refer to the first sentence after Eqn 7, and lines 4 and 10 of Algorithm 1.

V in Eqn 6 is the input to the TopK function, i.e., the representation of each missing variable.

E in Eqn 7 is the shared expert models which consists of E expert models.

Figure 2 is an example of a single $i_{r,t}$ in the CRW-Temp dataset, which contains the observations of 9 variables over the past 30 days ($\beta=30$). From left to right and top to bottom, the variables follow the same order as listed in lines 14 to 16 on page 14. The x-axis represents timestamps for all subplots, while the y axis specifies the values of each variable in a subplot. The units of the y-axis in different subplots are different, so we apply z-normalizaition on all variables before plotting the temporal trend image.

LLMs in Eqn 4 and 10 are GPT-3.5-turbo and LLaMA2-7b, respectively. We will modify the notation of LLM in Eqn 4 in the future version.

Eqn 10 in the algorithm is in line 13.

Linear layer after Eqn 10 is: $(Dropout(Q_{r, t}W_{1}+b_{1}))W_{2}+b_{2}$. It is used to transform the output representation from the LLM ($d=d_{LLM}$) to the final prediction ($d=1$).

$\eta_{1}$ and $\eta_{2}$ in the loss function are used to balance the regression loss and the imputation loss, which is much higher than the former one. The hyperparameters of $\eta_{1}$ and $\eta_{2}$ are decided through cross validation. This is a very common practice in literature (e.g., [1, 2]).

[1] Sener et al. “Multi-task learning as multi-objective optimization.” NeurIPS’18.

[2] Bhattacharjee et al. “Mult: An end-to-end multitask learning transformer.” CVPR’22.

Figure 3. From left to right, the first two are fixed settings while the second two are random settings.

Average RMSE increase ratio of the baselines is: $[(5.92-2.35)/2.35+(6.35-1.87)/1.87+(7.01-2.58)/2.58+(6.94-2.49)/2.49]/4=185$%. We will fix this typo.