This work presents a modular two layer architecture for recognizing isolated sign language video and converting gloss sequences into fluent natural language text. The proposed pipeline comprises: (1) glosslevel sign recognition using spatiotemporal feature extraction (I3D, EfficientNet, MobileNet) with lightweight classifiers, and (2) natural language smoothing via large language models (LLMs) using prompt engineering. Unlike prior claims of continuous translation, this paper explicitly focuses on isolated signs as a practical building block for assistive systems where segmentation is either manually provided or handled by external modules. Comprehensive evaluations on Brazilian Sign Language (Libras) datasets demonstrate high accuracy in isolated sign classification (F1 > 0.97 for I3D-RGB + LR). A critical analysis of near perfect AUC scores reveals dataset limitations that are openly discussed. For gloss to text conversion, we evaluate LLM smoothing on both clean and noise injected gloss sequences and report BLEU scores under realistic conditions. The proposed architecture is scalable, modular, and adaptable to other sign languages, advancing accessibility for deaf and hard of hearing communities.
[1] Khan, A., Jin, S., Lee, G. H., Arzu, G. E., Nguyen, T. N., Dang, L. M., Choi, W., Moon, H. (2025). Deep learning approaches for continuous sign language recognition: A comprehensive review. IEEE Access.
[2] Renz, K., Stache, N. C., Albanie, S., Varol, G. (2021). Sign language segmentation with temporal convolutional networks. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2135–2139). IEEE. https://doi.org/10.1109/ICASSP39728.2021.
[3] Taher, H. A., Zeebaree, S. R. (n.d.). A critical study of recent deep learning based continuous sign language recognition. The Review of Socionetwork Strategies.
[4] Papadimitriou, K., Potamianos, G., Sapountzaki, G., Goulas, T., Efthimiou, E., Fotinea, S. E., Maragos, P. (2025). Greek sign language recognition for an education platform. Universal Access in the Information Society, 24(1), 51– 68.
[5] Geetha, M., Aloysius, N., Somasundaran, D. A., Raghunath, A., Nedungadi, P. (2025). Toward real-time recognition of continuous Indian sign language: A multi-modal approach using RGB and pose. IEEE Access, 13, 60270–60283.
[6] Algafri, H., Luqman, H., Alyami, S., Laradji, I. (2025). SSLR: A semi-supervised learning method for isolated sign language recognition (arXiv:2504.16640). arXiv. https://arxiv.org/abs/2504.16640.
[7] Cerna, L. R., Cardenas, E. E., Miranda, D. G., Menotti, D., Camara-Chavez, G. (2021). A multimodal LIBRASUFOP Brazilian sign language dataset of minimal pairs using a Microsoft Kinect sensor. Expert Systems withApplications, 167, 114179. https://doi.org/10.1016/j.eswa.2020.114179.
[8] Renz, K., Stache, N. C., Albanie, S., Varol, G. (2021). Sign language segmentation with temporal convolutional networks. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2135–2139). IEEE.
[9] Wei, C., Zhao, J., Zhou, W., Li, H. (2020). Semantic boundary detection with reinforcement learning for continuous sign language recognition. IEEE Transactions on Circuits and Systems for Video Technology, 31(3), 1138–1149.
[10] Khan, S., Bailey, D. G., Gupta, G. S. (2014). Pause detection in continuous sign language. International Journal of Computer Applications in Technology, 50(1–2), 75–83.
[11] Yin, K., et al. (2021). Gloss2Text: Language model smoothing for sign language translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2021).
[12] Voskou, A., et al. (2022). BERT for sign language translation. In Proceedings of the IEEE Spoken Language Technology Workshop (SLT 2022).
[13] Camgoz, N. C., et al. (2018). Neural sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (p. 7784–7793).
[14] Tarrés, L., et al. (2023). Sign language translation from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (p. 5625–5635).
[15] Roy, P., et al. (2024). American sign language video-to-text translation. arXiv Preprint, arXiv:2402.07255.
[16] Roy, P., Han, J. E., Chouhan, S., Thumu, B. (2024). American sign language video to text translation. arXiv Preprint, arXiv:2402.07255.
[17] Tarrés, L., Gállego, G. I., Duarte, A., Torres, J., Giró-i-Nieto, X. (2023). Sign language translation from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (p. 5625–5635).
[18] Belhumeur, P. N., Hespanha, J. P., Kriegman, D. J. (1997). Eigenfaces vs. Fisherfaces: Recognition using classspecific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 711–720. https://doi.org/10.1109/34.598228
[19] Camgoz, N. C., Hadfield, S., Koller, O., Ney, H., Bowden, R. (2018). Neural sign language translation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (p. 7784–7793). IEEE.
[20] Cerna, L. R., Cardenas, E. E., Miranda, D. G., Menotti, D., Camara-Chavez, G. (2021). A multimodal LIBRASUFOP Brazilian sign language dataset of minimal pairs using a Microsoft Kinect sensor. Expert Systems with Applications, 167, 114179.
[21] Renz, K., Stache, N. C., Albanie, S., Varol, G. (2021). Sign language segmentation with temporal convolutional networks. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2135–2139). IEEE.
[22] Wei, C., Zhao, J., Zhou, W., Li, H. (2020). Semantic boundary detection with reinforcement learning for continuous sign language recognition. IEEE Transactions on Circuits and Systems for Video Technology, 31(3), 1138–1149.
[23] Bull, H., Gouiffès, M., Braffort, A. (2020). Automatic segmentation of sign language into subtitle-units. In European Conference on Computer Vision (p. 186–198). Springer.
[24] Khan, S., Bailey, D. G., Gupta, G. S. (2014). Pause detection in continuous sign language. International Journal of Computer Applications in Technology, 50(1–2), 75–83.