Reconstruction of optical properties of real scene objects from images by taking into account secondary illumination and selecting the most important points

Мұқаба

Дәйексөз келтіру

Толық мәтін

Ашық рұқсат Ашық рұқсат
Рұқсат жабық Рұқсат берілді
Рұқсат жабық Рұқсат ақылы немесе тек жазылушылар үшін

Аннотация

This paper presents a method for reconstructing the optical properties of objects in a real scene, based on a series of its images with the use of differentiable rendering. The main goal of this study is to develop an approach that enables the high-accuracy reconstruction of the optical characteristics of scene objects while minimizing the computational costs. Introduction considers the relevance of creating realistic models of virtual scenes for computer graphics, as well as their application in virtual reality, augmented reality, and animation. It is noted that, in order to achieve image realism, it is necessary to take into account the scene geometry, illumination parameters, and optical properties of objects. In this study, it is assumed that the scene geometry and light sources are known, and the main task is to reconstruct the optical properties of objects. Section 3 describes the main stages of the proposed approach. The first stage involves data preprocessing, during which the key image points characterized by high brightness and uniform distribution over scene objects are selected. This significantly reduces the amount of data required for optimization. Next, using numerical differentiation and backward ray tracing, luminance gradients are calculated based on the model parameters. The proposed algorithm takes into account both primary and secondary illumination, which improves the accuracy of reconstructing the optical characteristics of the scene. At the final stage, the parameters of the optical models are reconstructed using the ADAM method, improved with the Optuna library for automatic hyperparameter selection. Section 4 describes the experiments carried out on the Cornell Box scene. The result of reconstructing the optical properties is considered and the original and reconstructed luminances are compared. Certain limitations due to the duration of calculations and the sensitivity to data outliers are identified and discussed in detail. In Conclusions, the results are summarized and directions for further development are outlined, including the transfer of calculations to the GPU and the use of more complex models of optical properties to improve the accuracy and speed of the algorithm.

Толық мәтін

Рұқсат жабық

Авторлар туралы

S. Kupriyanov

National Research ITMO University

Хат алмасуға жауапты Автор.
Email: stasz776@gmail.com
ORCID iD: 0009-0006-8623-3578
Ресей, Kronverkskii pr. 49, St. Petersburg, 197101

I. Kinev

National Research ITMO University

Email: igorkinevitmo@gmail.com
ORCID iD: 0000-0003-2929-1203
Ресей, Kronverkskii pr. 49, St. Petersburg, 197101

Әдебиет тізімі

  1. Veach E. Robust Monte Carlo methods for light transport simulation, PhD Dissertation, Stanford University, 1998.
  2. Bogolepov D.K., Ulyanov D. GPU-optimized bidirectional path tracing, Proceedings of the 21th International.Conference Central Europe on Computer Graphics, Visualization, and Computer Vision, 2013, p. 15.
  3. Veach E., Guibas L.J. Metropolis light transport, Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques–SIGGRAPH’97, New York: ACM Press/Addison-Wesley, 1997, pp. 65–76. https://doi.org/10.1145/258734.258775
  4. Bitterli B., Jakob W., Novák J., Jarosz W. Reversible jump metropolis light transport using inverse mappings, ACM Trans. Graphics, 2017, vol. 37, no. 1, p. 1. https://doi.org/10.1145/3132704
  5. Gruson A., West R., Hachisuka T. Stratified Markov chain Monte Carlo light transport, Comput. Graphics Forum, 2020, vol. 39, no. 2, pp. 351–362. https://doi.org/10.1111/cgf.13935
  6. Jensen H.W. Global illumination using photon maps, Rendering Techniques ’96, Pueyo X. and Schröder P., Eds., Vienna: Springer, 1996, pp. 21–30. https://doi.org/10.1007/978-3-7091-7484-5_3
  7. Kato H., Beker D., Morariu M., Ando T., Matsuoka T., Kehl W., Gaidon A. Differentiable rendering: A survey, arXiv Preprint, 2015. https://doi.org/10.48550/arXiv.2006.12057
  8. Phong B.T. Illumination for computer generated pictures, Commun. ACM, 1975, vol. 18, no. 6, pp. 311–317. https://doi.org/10.1145/360825.360839
  9. Cook R.L., Torrance K.E. A reflectance model for computer graphics, ACM SIGGRAPH Computer Graphics, 1981, vol. 15, no. 3, pp. 307–316. https://doi.org/10.1145/965161.806819
  10. Burley B. Physically based shading at Disney, ACM Trans. Graphics, 2012, p. 7.
  11. Loper M.M., Black M.J. OpenDR: An approximate differentiable renderer, Computer Vision–ECCV 2014, Fleet D., Pajdla T., Schiele B., Tuytelaars T., Eds., Lecture Notes in Computer Science, Cham: Springer, 2014, vol. 8695, pp. 154–169. https://doi.org/10.1007/978-3-319-10584-0_11
  12. Kato H., Ushiku Yo., Harada T. Neural 3D mesh renderer, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, IEEE, 2018, pp. 3907–3916. https://doi.org/10.1109/cvpr.2018.00411
  13. Genova K., Cole F., Maschinot A., Sarna A., Vlasic D., Freeman W.T. Unsupervised training for 3D morphable model regression, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, IEEE, 2018, pp. 8377–8386. https://doi.org/10.1109/cvpr.2018.00874
  14. Rhodin H., Robertini N., Richardt Ch., Seidel H.-P., Theobalt Ch. A versatile scene model with differentiable visibility applied to generative pose estimation, 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, 2015, IEEE, 2015, pp. 765–773. https://doi.org/10.1109/iccv.2015.94
  15. Kajiya J.T. The rendering equation, ACM SIGGRAPH Computer Graphics, 1986, vol. 20, no. 4, pp. 143–150. https://doi.org/10.1145/15886.15902
  16. Li T.-M., Aittala M., Durand F., Lehtinen J. Differentiable Monte Carlo ray tracing through edge sampling, ACM Trans. Graphics, 2018, vol. 37, no. 6, p. 222. https://doi.org/10.1145/3272127.3275109
  17. Zhang Ch., Wu L., Zheng Ch., Gkioulekas I., Ramamoorthi R., Zhao Sh. A differential theory of radiative transfer, ACM Trans. Graphics, 2019, vol. 38, no. 6, p. 227. https://doi.org/10.1145/3355089.3356522
  18. Zhao Sh., Jakob W., Li T.-M. Physics-based differentiable rendering: From theory to implementation, SIGGRAPH 2020: ACM SIGGRAPH 2020 Courses, New York: Association for Computing Machinery, 2020, p. 14. https://doi.org/10.1145/3388769.3407454
  19. Nimier-David M., Vicini D., Zeltner T., Jakob W. Mitsuba 2: A retargetable forward and inverse renderer, ACM Trans. Graphics, 2019, vol. 38, no. 6, p. 203. https://doi.org/10.1145/3355089.3356498
  20. Sorokin M.I., Zhdanov D.D., Zhdanov A.D., Potemin I.S., Bogdanov N.N. Restoration of lighting parameters in mixed reality systems using convolutional neural network technology based on RGBD images, Program. Comput. Software, 2020, vol. 46, no. 3, pp. 207–216. https://doi.org/10.1134/s0361768820030093
  21. Kinev I.E., Kupriyanov S.I. Restoring optical properties of scene objects by differentiable rendering with optimization of most important point selection, Trudy konferentsii GrafiKon–2024 (Proc. Conf. GrafiKon–2024), 2024, pp. 179–193.
  22. Zhdanov D.D., Guskov K.S., Zhdanov A.D., Potemin I.S., Kulbako A.Yu., Alexandrov Yu.V., Lopatin A.V., Sokolov V.G. Using a federated approach to synthesize images of confidential scene models, Light Eng., 2024, vol. 32, no. 4, pp. 89–102. https://doi.org/10.33383/2024-017
  23. Optuna – A hyperparameter optimization framework. https://optuna.org
  24. Adam – PyToch 2.5 documentation. https://pytorch.org/docs/stable/generated/torch.optim.Adam.html

Қосымша файлдар

Қосымша файлдар
Әрекет
1. JATS XML
Жүктеу
2. Fig. 1. The complete process of restoring a real scene.

Жүктеу (1MB)
Метадеректер
3. Fig. 2. The process of reconstructing optical properties using point-based optimization.

Жүктеу (1MB)
Метадеректер
4. Fig. 3. Calculation of average errors for each object coefficient from different camera angles.

Жүктеу (1MB)
Метадеректер
5. Fig. 4. Changing the coefficients during optimization for each object in the scene. (a): the process of restoration from the initial coefficient 0.3 to the original 0.5. (b): the process of restoration from the initial coefficients 0.1, 0.9, 0.3, 0.7, 0.5 to the original 0.5. (c): the process of restoration from the initial coefficients 0.5 to 0.1, 0.9, 0.3, 0.7, 0.5.

Жүктеу (2MB)
Метадеректер
6. Fig. 5. Rendering results and comparison of the brightness of the back wall, first camera angle. Left: original image. Right: restored.

Жүктеу (2MB)
Метадеректер
7. Fig. 6. Rendering results and comparison of the brightness of the front wall of the parallelepiped, third camera angle. Left: original image. Right: restored.

Жүктеу (2MB)
Метадеректер

© Russian Academy of Sciences, 2025