Hi
I would like to manipulate blend shapes of a usdz file in VisionOS, but I can not figure out how I could do that. In SceneKit it was a piece of cake with SCNNode and morphs, but with Entity class I got stuck.
is it even possible? (I would be very disappointed if not)
RealityKit doesn't currently support blendshapes which is stated here: https://developer.apple.com/documentation/realitykit/validating-usd-files#Review-skeletons
Currently, it would only be possible to either use a custom geometry modifier to apply the vertex offsets interpolated over time through set_model_position_update or using a MeshResource update routine to transform the vertices. Most of my animations are procedural, so I haven't yet tried to get the blendshape offsets and weights from the USD file, which would then need to be uploaded through a uniform. There doesn't seem to have any exposed getter akin to jointTranforms.
https://developer.apple.com/documentation/realitykit/altering_realitykit_rendering_with_shader_functions https://developer.apple.com/documentation/realitykit/modifying-realitykit-rendering-using-custom-materials#Write-a-Geometry-Modifier https://developer.apple.com/videos/play/wwdc2021/10075 https://developer.apple.com/documentation/visionos/diorama
thank you! btw, I hope Apple solve this soon. It was so good and easy in SceneKit
Just to further clarify, custom materials are not actually supported in Vision OS, so you would need to do this through a shader graph. It is possible to run custom metal shaders through a compositor layer, but that doesn't integrate with RealityKit and requires a custom renderer.
RealityKit doesn't currently support blendshapes which is stated here: https://developer.apple.com/documentation/realitykit/validating-usd-files#Review-skeletons
Currently, it would only be possible to either use a custom geometry modifier to apply the vertex offsets interpolated over time through set_model_position_update or using a MeshResource update routine to transform the vertices. Most of my animations are procedural, so I haven't yet tried to get the blendshape offsets and weights from the USD file, which would then need to be uploaded through a uniform. There doesn't seem to have any exposed getter akin to jointTranforms.
https://developer.apple.com/documentation/realitykit/altering_realitykit_rendering_with_shader_functions https://developer.apple.com/documentation/realitykit/modifying-realitykit-rendering-using-custom-materials#Write-a-Geometry-Modifier https://developer.apple.com/videos/play/wwdc2021/10075 https://developer.apple.com/documentation/visionos/diorama
thank you! btw, I hope Apple solve this soon. It was so good and easy in SceneKit
Just to further clarify, custom materials are not actually supported in Vision OS, so you would need to do this through a shader graph. It is possible to run custom metal shaders through a compositor layer, but that doesn't integrate with RealityKit and requires a custom renderer.
