Object Centric Visual Question Answering for Robotic Assembly Inspection and Workflow Fault Diagnosis Tasks
Main Article Content
Abstract
The rapid evolution of modern manufacturing paradigms necessitates highly adaptable and intelligent robotic assembly inspection cells. Traditional machine vision systems rely on rigid, pre-programmed rules and spatial templates, which frequently fail when confronted with unstructured environments, diverse product portfolios, or unexpected anomalies. This paper proposes a novel framework centered on Object-Centric Visual Question Answering to address the growing demand for flexible and cognitive inspection methodologies. By treating the robotic cell as a dynamic environment where human operators can query the system using natural language, the proposed framework enables intuitive interaction and diagnostic transparency. Unlike standard global-feature visual question answering models, our approach parses visual scenes into discrete, object-centric representations. This allows the system to ground natural language queries to specific mechanical components, fasteners, and spatial relationships within the assembly workspace. The research details the architectural design, object discovery mechanisms, and cross-modal reasoning required to achieve high-fidelity inspection results. Extensive evaluations on a custom dataset of robotic assembly scenarios demonstrate that the object-centric approach significantly outperforms global feature baselines in terms of accuracy, interpretability, and robustness against occlusions. Ultimately, this framework paves the way for more collaborative, human-centric robotic manufacturing systems capable of complex reasoning and real-time fault diagnosis.
Downloads
Article Details
Section
This work is licensed under a Creative Commons Attribution 4.0 International License.
-
All published articles are open access, with copyright shared between the authors and the journal. The journal retains the exclusive right to first publication and identifies itself as original publisher.
-
Articles are made available under a Creative Commons CC BY‑NC‑ND 4.0 license, permitting users to copy and distribute the content for non‑commercial purposes only, without modification, provided proper credit is given to authors and JCPHS.
-
Redistribution or reuse must preserve the full citation details and include a link to the license. No derivatives, adaptations, or commercial uses are permitted without prior written permission.
References
1. Qu, W., Shao, Y., Meng, L., Huang, X., & Xiao, L. (2024). A conditional denoising diffusion probabilistic model for point cloud upsampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20786-20795).
2. Xiao, Q., & Zhou, K. (2026). MACD: Model-Aware Contrastive Decoding via Counterfactual Data. arXiv preprint arXiv:2602.01740.
3. Qu, W., Wang, J., Gong, Y., Huang, X., & Xiao, L. (2025). An end-to-end robust point cloud semantic segmentation network with single-step conditional diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 27325-27335).
4. Li, Q., Liu, Z., Luo, W., Luo, T., & Hou, C. (2026). Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory. arXiv preprint arXiv:2605.24602.
5. Xu, W., Huang, H., Gong, Y., Yu, L., Wu, Q., & Zhang, J. (2025). Hierarchical multi-prototype discrimination: Boosting support-query matching for few-shot segmentation. IEEE Transactions on Multimedia.
6. Yang, Y., Ma, X., Li, C., Zheng, Z., Zhang, Q., Huang, G., ... & Zhao, Q. (2021). Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 10299-10312.
7. Qi, Z., Yuan, Y., Ruan, X., Wang, S., Zhang, W., & Huang, Q. (2024). Collaborative debias strategy for temporal sentence grounding in video. IEEE Transactions on Circuits and Systems for Video Technology, 34(11), 10972-10986.
8. Qi, Z., Wang, S., Zhang, W., & Huang, Q. (2024). Uncertainty-boosted robust video activity anticipation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 7775-7792.
9. Zhang, Jiaquan, et al. "Learning global hypothesis space for enhancing synergistic reasoning chain." arXiv preprint arXiv:2602.09794 (2026).
10. Li, Z., Chen, Z., Qu, F., Wang, M., Zhao, Y., Zhang, K., & Peng, Y. (2024). Point resampling and ray transformation aid to editable NeRF models. arXiv preprint arXiv:2405.07306.
11. Liu, Y., Liu, H., Wang, H., Meng, F., & Liu, M. (2023). BCAN: Bidirectional correct attention network for cross-modal retrieval. IEEE Transactions on Neural Networks and Learning Systems, 35(10), 14247-14258.
12. Peng, Q., Xue, H., Wang, P., & Chen, C. (2026, March). Lifelong Domain Adaptive 3D Human Pose Estimation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 10, pp. 8358-8366).
13. Qu, D., & Ma, Y. (2025). Magnet-bn: markov-guided Bayesian neural networks for calibrated long-horizon sequence forecasting and community tracking. Mathematics, 13(17), 2740.
14. Zhao, Haoyu, et al. "LSTD: Long Short-Term Temporal Diffusion for Video Generation." IEEE Transactions on Multimedia (2026).
15. Chu, Y. B., Li, J., Jia, P., Cui, J., Zhang, R., Kang, X., Lv, M., & Zhang, S. (2021). Irf1- and Egr1-activated transcription plays a key role in macrophage polarization: A multiomics sequencing study with partial validation. International immunopharmacology, 99, 108072. https://doi.org/10.1016/j.intimp.2021.108072
16. Peng, Q., Zheng, C., & Chen, C. (2023). Source-free domain adaptive human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4826-4836).
17. Huang, H., Zhang, J., Zhang, J., Xu, J., & Wu, Q. (2020). Low-rank pairwise alignment bilinear network for few-shot fine-grained image classification. IEEE Transactions on Multimedia, 23, 1666-1680.
18. Guo, Zixin, Kai Zhao, and Luyan Zhang. "InstanceRSR: Real-World Super-Resolution via Instance-Aware Representation Alignment." ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2026.
19. Li, Q., Luo, T., & Liao, J. (2025). Theory-inspired deep multi-view multi-label learning with incomplete views and noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20706-20715).
20. Wang, M., Li, J., Li, Z., Luo, C., Chen, B., Xia, S. T., & Wang, Z. (2023, October). Unsupervised anomaly detection with local-sensitive VQVAE and global-sensitive transformers. In 2023 IEEE International Conference on Image Processing (ICIP) (pp. 1080-1084). IEEE.
21. Li, Z., Bai, X., Lu, J., Shen, P., Lam, E. Y., & Peng, Y. (2026). EventTracer: Fast Path Tracing-based Event Stream Rendering. IEEE Transactions on Visualization and Computer Graphics.
22. Ziyue Wang, Sheng Jin, Zhongrong Zuo, Jiawei Wu, Han Qiu, Qi She, Hao Zhang, and Xudong Jiang. Video-ktr: Re?inforcing video reasoning via key token attribution. arXiv preprint arXiv:2601.19686, 2026.
23. Zhao, Y., Li, Z., Guo, X., & Lu, Y. (2022). Alignment-guided temporal attention for video action recognition. Advances in Neural Information Processing Systems, 35, 13627-13639.
24. Zhou, D. (2026). AI-Driven Hybrid SAST–DAST–SCA–IAST Framework for Risk-Based Vulnerability Prioritization in Microservice Architectures.
25. Cong, P., Xu, Y., Ren, Y., Zhang, J., Xu, L., Wang, J., ... & Ma, Y. (2023, June). Weakly supervised 3d multi-person pose estimation for large-scale scenes based on monocular camera and single lidar. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 1, pp. 461-469).
26. Song, S., Tang, Y., & Qin, R. (2025). Synthetic Data Matters: Re-training with Geo-typical Synthetic Labels for Building Detection. IEEE Transactions on Geoscience and Remote Sensing.
27. Qu, D., Ma, Y., & Zhang, S. (2025, November). OAMF: Optics-Accelerated Multimodal Learning with Markov Temporal Priors and Fourier Regularization. In 2025 4th International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) (pp. 600-605). IEEE.
28. Tu, P., Huang, Y., Zheng, F., He, Z., Cao, L., & Shao, L. (2022, June). Guidedmix-net: Semi-supervised semantic segmentation by using labeled images as reference. In Proceedings of the AAAI conference on artificial intelligence (Vol. 36, No. 2, pp. 2379-2387).
29. Sang, Y. (2025, October). AutoCrit: A Meta-Reasoning Framework for Self-Critique and Iterative Error Correction in LLM Chains-of-Thought. In 2025 6th International Conference on Machine Learning and Computer Application (ICMLCA) (pp. 1177-1180). IEEE.
30. Huang, H., Zhang, C., Hu, Q., & Zhu, P. (2016, August). Multi-view representative and informative induced active learning. In Pacific Rim international conference on artificial intelligence (pp. 139-151). Cham: Springer International Publishing.
31. Lv, Q., Kong, W., Li, H., Zeng, J., Qiu, Z., Qu, D., ... & Pang, J. (2025). F1: A vision-language-action model bridging understanding and generation to actions. arXiv preprint arXiv:2509.06951.
32. Zhang, J., Zhang, J., Song, Z., Shi, Z., Zhao, C., Shi, Y., ... & Wang, J. (2024). Hoi-m^ 3: Capture multiple humans and objects interaction within contextual environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 516-526).
33. Ma, Y., & Qu, D. (2025, August). GEL-FMO: Gauge-Equivariant Liquid Fourier-Markov Operators for Uncertainty-Certified Multimodal Reasoning. In 2025 5th International Conference on Advanced Algorithms and Neural Networks (AANN) (pp. 604-607). IEEE.
34. Agrawal, H., Huang, X., Kim, J. H., Li, Y., Ma, Y., Ning, T., & Tao, Y. (2022). Automated image processing and insight presentation. Patent / invention record. Publication date: 2022-04-05.