I have spent the last several months at arguably one of the best places in the country to learn about 3D capture, and I have gotten an incredible amount out of that experience so far. When I started in February the position that I was interning for was fairly nebulous, initially being kind of a general position where I helped people out in the background and months later ending up in a bit of a developer and exploratory R&D role. This shift was due to the absolutely awesome level of autonomy that I was given, which initially arose from a day or two period early in the internship where everyone who was in the office was more or less busy and occupied and I didn't have anyone to shadow or explicitly tell me what to do. I took this time to start messing with a 3D javascript library called three.js to see what I could do with it, after I had a conversation previously with management about the lack of presentational materials when they do a deliverable hand-off to clients (mentioned in my last blog post). They saw what I was able to put together relatively quickly and asked if certain things were possible and for many of the things I told them I had no real idea and wouldn’t know until I tried, so I tried. They gave me the space to do so, so I just kept trying things and the culmination of that is a comprehensive viewer platform for various types of 3D data, with the company now planning on using it internally over the summer (at least).
Direct Dimensions works on 3D captures of very tiny things, huge outdoor scenes, engineering related things, cultural heritage related things, things that require exact precise measurement and things that are purely for visualization. This wide array of subjects and purposes makes it difficult to really convey what they do to the general public, but the gist is that they’re good at all of it and are genuinely capable of capturing the world for any application that you can think of. During the development of this application, I made sure to import and play with every data type that they had on hand in order to get an idea of what it would take to display that data reasonably well on the web. The big three are 3D mesh models (with multiple distinct types), point clouds from lidar or laser scans, and gaussian splats. There are platforms that handle each of these, but none that handle all reasonably well in an open, non locked-down manner. I was able to compile notes from a ton of reading and hammer down a workflow for each type that used the best possible techniques and technology in terms of compression, chunked streaming and level of detail loading in order to get a smooth user experience for each that could display well on even modest computers or phones while not sacrificing detail. I looked at many other programs and basically took the best ideas from each while fixing the things I felt were not good or impeded the user experience. They like the application enough that they asked me if I could integrate it with one of their automated scanner products, and I have heard them make clients aware of its existence (which bodes very well for my potential future at the company).
I ended up becoming knowledgeable about many file types and capture methods and now have an opinion on what $50,000 lidar scanner is best (it's wild to me that I got hands on one). I have played with a bunch of programs for processing 3D data and now have a comprehensive understanding of the current landscape. I can not truly state how valuable this internship has been, and I hope that I can give back in a meaningful way. Aside from the software that I have put together I have also been experimenting with scene reconstruction and gaussian splats in particular, which are a bit of a frontier in terms of the tech and is a space that is progressing and moving incredibly quickly. I have been paying attention to academic papers that have been coming out and try to be the one that makes the rest of the company aware of the latest and greatest, and I believe that in the coming years there will be a distinct shift away from creating 3D content with very rigidly defined manipulations of XYZ coordinates to more so growing and evolving 3D content from carefully prepared seed data. Basically this will make a lot of the vision-oriented (as opposed to metrology or archival) use cases for 3D scan data affordable and workable, and we will be able to use generative techniques to repair and augment scan data in new ways. One example would be taking a 3D laser scan of an old or damaged building to get accurate base geometry and then pairing it with photos of what the building looked like 100 years ago in order to create an accurate 3D model of the building as it once was. You would be combining the real with the inferred, but the data is all there so nothing would have to be speculatively created from assumptions. What someone would previously have to meticulously model by hand could be done reasonably quickly (under supervision) to create an asset that could be used in a museum or in a virtual experience. It’s something I find very exciting, and I am lucky to be this close to a technological frontier. All in all I am incredibly grateful for the experience I have had here over this semester and can not wait to continue the work that I have started.
Below is another screenshot of the web application that I put together, this time of the editor/scene composition interface which shows a lidar scan of the Direct Dimensions building that I was present during the capture of. Usually point clouds like these are locked into difficult to use and heavy desktop applications.
No comments:
Post a Comment