Delving into LLMs’ visual understanding ability using SVG to bridge image and text

Dear Reviewer oVfB,

We are grateful for your acknowledgment of our study’s novelty, and comprehensive evaluation of both discriminative and generative visual understanding of LLMs. For your questions—we'll make revisions accordingly.

“The authors should evaluate more Multimodal LLMs such as BLIP-2, InstructBLIP, MiniGPT-4, mPLUG-Owl, etc”

We appreciate your suggestion. We have expanded our comparison in the table to include additional Multimodal LLMs. The updated results indicate that combining SVG with LLMs outperforms other models like BLIP-2, InstructBLIP, MiniGPT-4, and mPLUG-Owl in handling complex, structured reasoning tasks.

Caption: Performance comparison across different models on the Sort-of-Clever Dataset.

“How does the LLM perform with SVG representations on more complicated images?”

Thank you for raising this concern. We agree that the current approach of integrating SVG with LLMs for complex visual tasks may fall short, primarily due to the loss of fine details in converting and the lengthy sequences of encoding extensive details. To tackle this, a potential solution is to leverage large vision models to extract the essence of images and convert that essence into the SVG format for LLM processing. An example is utilizing SAM [1] for object mask segmentation, which is then translated to SVG for LLMs, as depicted in Figure 6 (d). We then show a recognition result in Figure 7 (b).

[1] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao et al. "Segment anything." ICCV (2023).

Dear reviewer oVfB,

Thank you for reviewing our work to enhance the quality of the paper! Has our rebuttal adequately addressed your concerns? If you still have any issues with our rebuttal or if there are any new concerns, we are more than willing to continue the discussion with you.

"Whether the format, sequence, or other characteristics of textual descriptions in Scalable Vector Graphics (SVG) will affect their subsequent task performance"

and

"What could be the potential impact if a minor percentage of errors into the representation of SVGs is introduced?"

Thanks for both of your suggestions. To rigorously evaluate the robustness of LLMs against variations in SVG data, we conducted three distinct experiments, where we: (i) shuffle the order of paths, (ii) randomize path coordinate replacement, and (iii) randomize string replacement. Each experiment was designed to mimic real scenarios of imperfections in SVG data, providing insights into using SVG with LLMs under challenging conditions.

(i) Path Shuffle Experiment SVG data consists of multiple path elements, each representing an object or line in the image. In this experiment, we tested the model's ability to interpret hand-drawn SVG data from the MNIST dataset when the sequence of path elements was shuffled. Note that, by the very nature of SVG data, this alteration will not change the final image; and hence, it is important to test if the LLM can remain invariant to this change, even though we have not explicitly provided it with this knowledge. The goal is to assess the model's ability to correctly interpret the digit, regardless of the order of path elements. The results are summarized in the table below. Results indicate that SVG is robust to path shuffle at test time, maintaining a comparable performance compared to the SVG representation with the original path ordering.

Caption: Model accuracy for the path shuffle experiment.

(ii) Random Path Coordinate Replacement We follow the reviewer's suggestion and introduce a subtle form of noise by randomly altering the coordinates in the SVG path data. Each numerical value within the path commands is randomly translated within a specific range $k$. For example, for $k=1$, ... <path d='M0 0 C18 0 17...'> might become ... <path d='M1 0 C18 1 18...'>. We test several variations: (i) a minor adjustment within a 1/28th range (reflecting the 28x28 resolution of MNIST) and (ii) a more significant alteration within a 5/28th range. This simulates real-world scenarios of minor inaccuracies in SVG data, such as those resulting from conversion errors or imprecise digitization. Table shown below presents the accuracy under different noise scales. Results indicate data SVG representation is decently robust to the perturbation of the coordinate values.

Caption: Model accuracy under different levels of coordinate noise.

(iii) Random String Replacement Finally, we design the most aggressive test of robustness, where we replace random characters in the SVG strings with any English alphabet letter, digit, or special symbol, regardless of whether they are numerical values or part of SVG commands (like transform='translate(0,0)'). The experiment is conducted with varying probabilities for character replacement, as shown below. Surprisingly, even after replacing 20% of a string with random characters, SVG with LLMs maintains a high accuracy rate of 90.79%. This suggests that SVG with LLMs can handle a wide range of perturbations in SVG data.

The following table presents the accuracy with varying probabilities of random string replacement:

Caption: Accuracy with different probabilities of random string replacement.

In conclusion, the results from these experiments demonstrate the robustness of using SVG data with LLMs for visual understanding. Despite the introduction of perturbations, using SVG data with LLMs can perform well under challenging conditions.

Dear Reviewer Vo5P,

We are grateful for your acknowledgment of our study’s novelty, comprehensive evaluation, and promising results. For your questions; we'll make revisions accordingly.

"How well do SVG representations capture complex images? Could you provide instances or examples where this approach has limitations?"

Thank you for pointing out this concern. We acknowledge the limitations of using SVG with LLMs for complex visual tasks, due to the loss of fine-grained details during conversion and the challenge of managing prohibitively long sequences when encoding detailed photographic content. To tackle this, a potential solution is to leverage large vision models to extract the essence of images and convert that essence into the SVG format for LLM processing. An example is utilizing SAM [1] for object contour segmentation, which is then translated to SVG for LLMs, as depicted in Figure 6 (d). We then show a recognition result in Figure 7 (b).

"A comparison of LLMs' performance using SVGs with traditional vision-based models could have provided more context about the relative effectiveness of this approach."

Thank you for your suggestion. We'd like to clarify that in our paper, we have compared the performance of LLMs using SVGs with traditional vision-based models. This includes:

• Visual reasoning tasks in Table 1, where LLMs with SVGs outperform SoTA vision-based models in both in-distribution and out-of-distribution scenarios.

• Image classification tasks in Table 2, showing LLMs with SVGs again surpassing SoTA vision-based models under similar conditions.

• Visual prompting in Table 3, where LLMs with SVGs demonstrate superior performance compared to SoTA vision-based models.

[1] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao et al. "Segment anything." ICCV (2023).

Dear reviewer Vo5P,

Thank you for reviewing our work to enhance the quality of the paper! Has our rebuttal adequately addressed your concerns? If you still have any issues with our rebuttal or if there are any new concerns, we are more than willing to continue the discussion with you.

“The analysis presented in this work is constrained by its reliance on existing methodologies. The absence of innovative input representations, model adaptations, or new evaluation benchmarks suggests a missed opportunity for advancing the field.”

We recognize that while our study relies on established techniques, these techniques are sufficiently effective for practical use in simple scenarios. We agree that addressing complex real-world images may necessitate the development of new methods. We are grateful for being the pioneers to start the investigation of LLMs for understanding and reasoning about images without relying on visual data, and we are committed to advancing our research to tackle real-world challenges more effectively.

Dear Reviewer uhQW,

We are grateful for your acknowledgment of our study’s comprehensive evaluation, and demonstration of the robustness of the LLM + SVG under distribution shift. Below, we respond to your specific questions.

“While the experiments are detailed, the novelty of the individual components used within the work is unclear”

We wish to clarify that our study primarily aims to provide a thorough analysis of the strengths and weaknesses of employing LLMs to understand and reason about images without visual data. Below, we detail the novelty of each component in our study.

• Comprehensive study of LLMs across various visual tasks using SVG format, including visual reasoning, image classification, and image generation.
• Investigation into LLMs' performance in visual data using SVG format under conditions of distribution shifts, expanding our understanding of their robustness.
• Analysis of LLMs' generative capabilities in image generation and editing based on interactive feedback within the SVG format.

“The comparative results between LLM+SVG and CNN+PNG in the first experiment are intriguing but lack a clear explanation. ... isolate the impact of each component.”

Thank you for your suggestion. Following your advice, we conducted additional experiments to assess the impact of different input formats using the same model, as shown in the table below. Specifically, we compare the SVG input format with GPT4 against PNG input format with GPT4V (likely larger than GPT4 due to its visual encoding network component) for a balanced evaluation. The results indicate that using SVG with GPT4 significantly outperforms using PNG with GPT4V, in complex, structured reasoning tasks.

Caption: Performance comparison across different models on the Sort-of-Clever Dataset.

Additionally, to ensure a fair comparison, we compared the performance of both SVG and PNG inputs using the same Vicuna model for image classification in the table below As shown in the following figure, under the same model architecture, results indicate that SVG is more robust than rasterized representation.

Caption: Performance comparison across different models on the MNIST Dataset.

“The chosen dataset and task do not seem to reflect complex real-world scenarios and may inherently favor SVG representations due to their structured nature.”

We thank the reviewer for raising and highlighting an important point. First, we would like to re-emphasize the main goal of our work: to see how much knowledge LLMs already have in understanding visual data, even though one might not naturally assume them to be suited for any vision task. Therefore, any experiments that we conduct (e.g., Table 1) should be seen as a way to study that question, and not necessarily as a way to demonstrate that LLMs are fundamentally superior to CNN/transformers. And since, to the best of our knowledge, our work is the first to thoroughly study this hidden capability of LLMs, we have chosen the simpler settings. Our hope, however, is that our work can motivate investigations into other, more complex, vision tasks that can be solved with the help of an LLM.

Dear reviewer uhQW,

Thank you for reviewing our work to enhance the quality of the paper! Has our rebuttal adequately addressed your concerns? If you still have any issues with our rebuttal or if there are any new concerns, we are more than willing to continue the discussion with you.

Glad that we have addressed your aforementioned concerns!

Your suggestion to evaluate using SVG with LLMs on more real-world datasets is greatly appreciated. We further studied the limitations of using SVG with LLMs in handling complex real-world tasks in the following.

1)

Due to limited computational resources, we performed experiments involving a subset of ImageNet. Specifically, we conduct experiments on Imagenette, which is composed of images from 10 classes in ImageNet. We achieved 68.14% top-1 accuracy on the corresponding 10-class test set after training with SVG format using Vicuna-7B model, whereas a ResNet50 model receives 90.62% accuracy on the test set after trained with 200 epochs.

Our initial results suggest that LLMs can somehow understand the semantics of the real-world image by leveraging SVG, yet lag behind raster representations using the vision model architectures like ResNet with a clear gap. We believe it's mainly because of the loss of fine-grained details during SVG conversion. We have added your suggested experiments to our paper to further underscore the limitation of using SVG with LLMs in understanding complicated visual tasks, and we are committed to expanding our experiments with more categories and varied datasets in the final version.

2)

Additionally, we agree we’ve selected some datasets/tasks that might favor SVG, but we observed that even with the advanced large vision model ConvNext, performance on the simple CMNIST dataset (Table 2) deteriorates significantly under out-of-distribution conditions. Utilizing the SVG format, in contrast, might offer a complementary solution to large vision models for better handling of such out-of-distribution scenarios.