Render advanced 3D graphics and perform data-parallel computations using graphics processors using Metal.

Metal Documentation

Posts under Metal tag

281 Posts
Sort by:
Post not yet marked as solved
4 Replies
204 Views
I am working on an application where we are planning to use Metal for directly rendering custom content. When user looks at something on the rendered image, I want to get the position or ray of cursor (the point where the user is currently looking at) to render something else like a crosshair. Is it possible to get the cursor position information on VisionOS to accomplish this? How can I know if something is being hovered on by the eyes?
Posted Last updated
.
Post not yet marked as solved
1 Replies
125 Views
Platform 'METAL' is experimental and not all JAX functionality may be correctly supported! 2024-03-23 22:04:38.947506: W pjrt_plugin/src/mps_client.cc:563] WARNING: JAX Apple GPU support is experimental and not all JAX functionality is correctly supported! Metal device set to: Apple M1 Pro systemMemory: 16.00 GB maxCacheSize: 5.33 GB loc("-":0:0): error: current mps dialect version is 1.0.0, can't parse version 1.1.0 /AppleInternal/Library/BuildRoots/495c257e-668e-11ee-93ce-926038f30c31/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphExecutable.mm:1097: failed assertion `Error importing MLIR bytecode. ' zsh: abort python -c 'import jax; print(jax.numpy.arange(10))'
Posted Last updated
.
Post marked as solved
1 Replies
115 Views
I tried to render to two layers using vertex amplification in my mesh shader program, but in the vision pro only the left eye has content and it contains 2 eyes image, and when I switch the mapping0.renderTargetArrayIndexOffset in the encoder, it does not transfer the image to the right eye. Using vertex amplification achieve 2 eyes rendering?
Posted
by Suan.
Last updated
.
Post marked as solved
1 Replies
111 Views
I've been trying to find a C/C++ framework for apps on macos. I couldn't find good docs on metal. Is there a way to write C++ apps without any other library?
Posted Last updated
.
Post not yet marked as solved
0 Replies
85 Views
in Swift languange, CVMetalTextureCacheCreateTextureFromImage return CVMetalTexture, and CVMetalTexture is Swift class, so. it doesn't need to call CVBufferRelease manually. My question is : should I use a variable to keep strong reference until GPU finished (until addCompleteHandler callback ) ? cvmetaltexturecachecreatetexture
Posted
by ZoGo996.
Last updated
.
Post not yet marked as solved
0 Replies
96 Views
Currently (jax-metal 0.0.5) doesn't support matrix decompositions like Cholesky, LU, or Eigen, on Metal. I have a 64GB M1 Max on Sonoma 14.2.1 (23C71). Eigen raises a NotImplemented error, but Cholesky and jax.linalg.inv error out. When using the CPU by setting export JAX_PLATFORM_NAME=CPU then the decompositions work. The error I am getting: XlaRuntimeError Traceback (most recent call last) Cell In[33], line 1 ----> 1 jsl.cho_factor(Sigma_0_inv) File ~/miniconda3/envs/jaxmetal/lib/python3.10/site-packages/jax/_src/scipy/linalg.py:61, in cho_factor(failed resolving arguments) 56 @_wraps(scipy.linalg.cho_factor, 57 lax_description=_no_overwrite_and_chkfinite_doc, skip_params=('overwrite_a', 'check_finite')) 58 def cho_factor(a: ArrayLike, lower: bool = False, overwrite_a: bool = False, 59 check_finite: bool = True) -> tuple[Array, bool]: 60 del overwrite_a, check_finite # Unused ---> 61 return (cholesky(a, lower=lower), lower) File ~/miniconda3/envs/jaxmetal/lib/python3.10/site-packages/jax/_src/scipy/linalg.py:54, in cholesky(failed resolving arguments) 49 @_wraps(scipy.linalg.cholesky, 50 lax_description=_no_overwrite_and_chkfinite_doc, skip_params=('overwrite_a', 'check_finite')) 51 def cholesky(a: ArrayLike, lower: bool = False, overwrite_a: bool = False, 52 check_finite: bool = True) -> Array: 53 del overwrite_a, check_finite # Unused ---> 54 return _cholesky(a, lower) File ~/miniconda3/envs/jaxmetal/lib/python3.10/site-packages/jax/_src/compiler.py:255, in backend_compile(backend, module, options, host_callbacks) 250 return backend.compile(built_c, compile_options=options, 251 host_callbacks=host_callbacks) 252 # Some backends don't have host_callbacks option yet 253 # TODO(sharadmv): remove this fallback when all backends allow compile 254 # to take in host_callbacks --> 255 return backend.compile(built_c, compile_options=options) XlaRuntimeError: UNKNOWN: /var/folders/9d/0035yr7j3bx84h3ghpp_86pc0000gn/T/ipykernel_40684/3046650730.py:1:0: error: failed to legalize operation 'mhlo.cholesky' /var/folders/9d/0035yr7j3bx84h3ghpp_86pc0000gn/T/ipykernel_40684/3046650730.py:1:0: note: see current operation: %5 = "mhlo.cholesky"(%4) {lower = true} : (tensor<200x200xf32>) -> tensor<200x200xf32>
Posted Last updated
.
Post not yet marked as solved
7 Replies
221 Views
I wanna draw a pixel buffer directly on the screen with the Metal API. in OpenGL I can use glDrawPixels how to do it in Metal?
Posted
by Key-Real.
Last updated
.
Post not yet marked as solved
3 Replies
842 Views
I'm using DrawableQueue to create textures that I apply to my ShaderGraphMaterial texture. My metal render is using a range of alpha values as a test. My objects displayed with the DrawableQueue texture are working as expected, but the alpha component is not working. Is this an issue with my DrawableQueue descriptor? My ShaderGraphMaterial? A missing setting on my scene objects? or some limitation in visionOS? DrawableQueue descriptor let descriptor = await TextureResource.DrawableQueue.Descriptor( pixelFormat: .rgba8Unorm, width: textureResource!.width, height: textureResource!.height, usage: [.renderTarget, .shaderRead, .shaderWrite], // Usage should match the requirements for how the texture will be used //usage: [.renderTarget], // Usage should match the requirements for how the texture will be used mipmapsMode: .none // Assuming no mipmaps are needed for the text texture ) let queue = try await TextureResource.DrawableQueue(descriptor) queue.allowsNextDrawableTimeout = true await textureResource!.replace(withDrawables: queue) Draw frame: guard let drawable = try? drawableQueue!.nextDrawable(), let commandBuffer = commandQueue?.makeCommandBuffer()//, //let renderPipelineState = renderPipelineState else { return } let renderPassDescriptor = MTLRenderPassDescriptor() renderPassDescriptor.colorAttachments[0].texture = drawable.texture renderPassDescriptor.colorAttachments[0].loadAction = .clear renderPassDescriptor.colorAttachments[0].storeAction = .store renderPassDescriptor.colorAttachments[0].clearColor = clearColor /*renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColor( red: clearColor.red, green: clearColor.green, blue: clearColor.blue, alpha: 0.5 )*/ renderPassDescriptor.renderTargetHeight = drawable.texture.height renderPassDescriptor.renderTargetWidth = drawable.texture.width guard let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor) else { return } renderEncoder.pushDebugGroup("DrawNextFrameWithColor") //renderEncoder.setRenderPipelineState(renderPipelineState) // No need to create a render command encoder with shaders, as we are only clearing the drawable. // Since we are just clearing the drawable to a solid color, no need to draw primitives renderEncoder.endEncoding() commandBuffer.commit() commandBuffer.waitUntilCompleted() drawable.present() }
Posted Last updated
.
Post not yet marked as solved
0 Replies
127 Views
On macOS 14.2.1 Sonoma, when our app is displayed in full screen, there is a problem where the operations at hand are not reflected at all on the connection destination. This issue does not occur on macOS 13 Ventura. If you stop displaying full screen and reduce the window size, you will be able to operate it to some extent. Were there any specifications or technical changes around the screen between Ventura and Sonoma? Mac at hand* - Relay server - Windows to connect to *This does not occur on windows, android, or ios.
Posted Last updated
.
Post not yet marked as solved
0 Replies
180 Views
With the advent of the third dimension, I wanted to know wether if it's currently possible to display the flat swiftUI Views with some thickness in xrOS? While the .frame(depth: CGFloat?) does the job for Views in general, I am eager for a more granular level of control at the pixel-specific level. I was hoping that there are lower level APIs to achieve this & I've looked into the fairly new layerEffect shader API, yet it seems it's incapable of setting the depths of pixels...
Posted
by Treata.
Last updated
.
Post not yet marked as solved
9 Replies
1.8k Views
I'm trying to debug my metal shaders in Xcode 14.2. However clicking "Capture metal GPU" while debugging recently started showing the following error: Capturing MTLPipelineLibrary is not supported. Unsupported method: -[MTLDevice newPipelineLibraryWithFilePath:error:] To enable capturing, disable calls to unsupported APIs and relaunch your application. I can't find any info about MTLPipelineLibrary or how to disable it. I've also confirmed that Metal GPU Frame Capture is enabled in my build What's causing this issue and how can I work around it so I can debug my shaders again?
Posted Last updated
.
Post not yet marked as solved
1 Replies
334 Views
Hi, I'm displaying linear gray by CAMetalLayer with the shader below. fragment float4 fragmentShader(VertexOut in [[stage_in]], texture2d<float, access::sample> BGRATexture [[ texture(0) ]]) { float color = in.texCoordinates.x; return float4(float3(color), 1.0); } And my CAMetalLayer has been set to linearSRGB. metalLayer.colorspace = CGColorSpace(name: CGColorSpace.linearSRGB) metalLayer.pixelFormat = .bgra8Unorm Why the display seem add gamma? Apparently the middle gray is 187 but not 128.
Posted
by fsjack.
Last updated
.
Post marked as solved
1 Replies
219 Views
I'm currently using Metal to create a game board with floating balloons; each balloon is a SKSpriteNode with an image of a balloon attached. The user touches and drags the balloon to a second balloon, merging the two. Exactly how they get merged is based on input from the user. I have a UISegmentedControl that pops up where the user selects one of four responses and the merge occurs. Currently, the UISegmentedControl pops up in the middle of the game board; however, I would like it to overlay on top of the first balloon instead. I have tried this once the two balloons touch each other: bubble1.physicsBody!.velocity = CGVector(dx: 0, dy: 0) // Stopping the balloon requestView.frame.origin.x = bubble1.position.x requestView.frame.origin.y = bubble1.position.y Where requestView is a UIView (with the same dimensions of the balloon) with various subviews (including the UISegmentedControl) and bubble1 is the SKSpriteNode (balloon). However, when I add the requestView as a subview of the game board, it does not overlay on top of the SKSpriteNode (bubble1). In fact, each time I try it, it doesn't even seem to appear in the same space relative to the location of the bubble1. Any thoughts on what I might be doing wrong? Thanks!
Posted Last updated
.
Post not yet marked as solved
0 Replies
222 Views
Hello, I have been following the excellent/informative "Metal for Machine Learning" from WWDC19 to learn how to do on device training (I have a specific use case for this) and it is all working really well using the MPSNNGraph. However, I would like to call my own metal compute/render function/pipeline to transform the inference result before calculating the loss, does anyone know if this possible and what would this look like in code? Please see my current code below, at the comment I need to call an intermediate compute/render function to transform the inference result image before passing to the MPSNNForwardLossNode. let rgbImageNode = MPSNNImageNode(handle: nil) let inferGraph = makeInferenceGraph() let reshape = MPSNNReshapeNode(source: inferGraph.resultImage, resultWidth: 64, resultHeight: 64, resultFeatureChannels: 4) //Need to call render or compute pipeline to post process in the inference result image let rgbLoss = MPSNNForwardLossNode(source:reshape.resultImage, labels:rgbImageNode, lossDescriptor:lossDescriptor) let initGrad = MPSNNInitialGradientNode(source:rgbLoss.resultImage) let gradNodes = initGrad.trainingGraph(withSourceGradient:nil, nodeHandler:nil) guard let trainGraph = MPSNNGraph(device: device, resultImage: gradNodes![0].resultImage, resultImageIsNeeded: true) else{ fatalError("Unable to get training graph.") } Thanks
Posted
by IainA.
Last updated
.
Post not yet marked as solved
0 Replies
172 Views
Hi, I am looking for a routine to perform complex-valued linear algebra on the GPU in python for scientific programming, in particular quantum physics simulations. At the moment I am looking for a routine for complex-valued matrix multiplication. I found MLX has a routine for float matrix multiplication, but it does not directly work for complex-valued matrices. I figured a work-around by splitting the complex valued matrix into real and imaginary part and working with the pair, but it makes it cumbersome to integrate with the remainder of the code. I was hoping for a library-based implementation similar to cupy. I also tried out using the tensorflow linear algebra routines, but I couldn't get them to run on the GPU by now. Specifically, a testfile with a tensorflow.keras.applications.ResNet50 routine runs on the GPU, but the routines from tensorflow.linalg and tensorflow.math that I tested (matmul, expm, eigh) were not running on the GPU. Any advice on how to make linear algebra calculations on mac GPUs work is highly appreciated! For my application the unified memory might be especially beneficial. Thank you!
Posted
by MG607.
Last updated
.
Post not yet marked as solved
1 Replies
202 Views
I used metal and CompositorLayer to render an immersive space skybox. In this space, the window created by the Swift UI I created only displays the gray frosted glass background effect (it seems to ignore the metal-rendered skybox and only samples and displays the black background). why is that? Is there any solution to display the normal frosted glass background? Thank you very much!
Posted
by zane1024.
Last updated
.
Post not yet marked as solved
0 Replies
188 Views
Hi We have an issue where the blurred background shown behind the Views is incorrect. We are using the compositor layer to render a fully immersive scene using Metal, so there is no real world passthrough. We have a couple of SwiftUI Views in the scene that have translucency and the glassBackground effect enabled. We expected that the background would show the blurred immersive scene, in the same way it shows the blurred real world in passthrough mode. Instead, it seems to still show the blurred real world, which feels strange from the user perspective. This is only visible on-device, in the simulator the background is always black. Thanks Mark
Posted Last updated
.
Post not yet marked as solved
2 Replies
249 Views
In the below code I have extracted face mesh vertices from ARKit face anchors and created a custom face mesh using SceneKit SCNGeometry. This enabled me to stretch face mesh vertices as per my requirement. Now the problem I am facing is as follows. I am trying to apply a lipstick texture material which is of type SCNMaterial. Although ARSCNFaceGeometry lets me apply different textures through SCNMaterial and SCNNode, I am not able to do the same using mu CustomFaceGeometry. When I am applying a lipstick texture which looks like the image attached below, the full face is getting colored or modified, I want only that part of the face which has texture transparency as >0 and I dont want other part of the face to be modified. Can you give me a detailed solution using code? // ViewController.swift import UIKit import ARKit import SceneKit import simd class ViewController: UIViewController, ARSCNViewDelegate, ARSessionDelegate{ @IBOutlet weak var sceneView: ARSCNView! let vertexIndicesOfInterest = [250] var customFaceGeometry: CustomFaceGeometry! var scnFaceGeometry: SCNGeometry! private var faceUvGenerator: FaceTextureGenerator! var faceGeometry: ARSCNFaceGeometry! override func viewDidLoad() { super.viewDidLoad() sceneView.delegate = self override func viewWillAppear(_ animated: Bool) { super.viewWillAppear(animated) let configuration = ARFaceTrackingConfiguration() sceneView.session.run(configuration) } } extension ViewController { func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) { guard let faceAnchor = anchor as? ARFaceAnchor else { return } customFaceGeometry = CustomFaceGeometry(fromFaceAnchor: faceAnchor) let customGeometryNode = SCNNode(geometry: customFaceGeometry.geometry) customFaceGeometry.geometry.firstMaterial?.fillMode = .lines customFaceGeometry.geometry.firstMaterial?.transparency = 0.0 customFaceGeometry.geometry.firstMaterial?.isDoubleSided = true node.addChildNode(customGeometryNode) } func renderer(_ renderer: SCNSceneRenderer, willUpdate node: SCNNode, for anchor: ARAnchor) { guard let faceAnchor = anchor as? ARFaceAnchor, let faceMeshNode = node.childNodes.first else { return } DispatchQueue.main.async { self.customFaceGeometry.update(withFaceAnchor: faceAnchor, node: faceMeshNode) } } } class CustomFaceGeometry { var geometry: SCNGeometry let lipImage = UIImage(named: "Face.scnassets/lip_arks_y7.png") init(fromFaceAnchor faceAnchor: ARFaceAnchor) { self.geometry = CustomFaceGeometry.createCustomSCNGeometry(from: faceAnchor)! } static func createCustomFaceGeometry(fromVertices vertices_o: [SCNVector3]) -> SCNGeometry { var vertices = vertices_o let vertexData = Data(bytes: vertices, count: vertices.count * MemoryLayout<SCNVector3>.size) let vertexSource = SCNGeometrySource(data: vertexData, semantic: .vertex, vectorCount: vertices.count, usesFloatComponents: true, componentsPerVector: 3, bytesPerComponent: MemoryLayout<Float>.size, dataOffset: 0, dataStride: MemoryLayout<SCNVector3>.stride) let indices: [Int32] = Array(0..<Int32(vertices.count)) let indexData = Data(bytes: indices, count: indices.count * MemoryLayout<Int32>.size) let element = SCNGeometryElement(data: indexData, primitiveType: .point, primitiveCount: vertices.count, bytesPerIndex: MemoryLayout<Int32>.size) return SCNGeometry(sources: [vertexSource], elements: [element]) } static func createGeometry(fromFaceAnchor faceAnchor: ARFaceAnchor) -> SCNGeometry let vertices = faceAnchor.geometry.vertices.map { SCNVector3($0.x, $0.y, $0.z) } return CustomFaceGeometry.createCustomFaceGeometry(fromVertices: vertices) } func update(withFaceAnchor faceAnchor: ARFaceAnchor, node: SCNNode) { if let newGeometry = CustomFaceGeometry.createCustomSCNGeometry(from: faceAnchor) { node.geometry = newGeometry let lipstickNode = SCNNode(geometry: newGeometry) let lipstickTextureMaterial = SCNMaterial() lipstickTextureMaterial.diffuse.contents = lipImage lipstickTextureMaterial.transparency = 1.0 lipstickNode.geometry?.firstMaterial = lipstickTextureMaterial node.geometry?.firstMaterial?.fillMode = .lines node.geometry?.firstMaterial?.transparency = 0.5 } } static func createCustomSCNGeometry(from faceAnchor: ARFaceAnchor) -> SCNGeometry? { let faceGeometry = faceAnchor.geometry var vertices: [SCNVector3] = faceGeometry.vertices.map { SCNVector3($0.x, $0.y, $0.z) } print(vertices[250]) let ll_ratio_y = Float(0.969999) vertices[290] = SCNVector3(x: vertices[290].x, y: vertices[290].y*ll_ratio_y, z: vertices[290].z) vertices[274] = SCNVector3(x: vertices[274].x, y: vertices[274].y*ll_ratio_y, z: vertices[274].z) vertices[265] = SCNVector3(x: vertices[265].x, y: vertices[265].y*ll_ratio_y, z: vertices[265].z) vertices[700] = SCNVector3(x: vertices[700].x, y: vertices[700].y*ll_ratio_y, z: vertices[700].z) vertices[730] = SCNVector3(x: vertices[730].x, y: vertices[730].y*ll_ratio_y, z: vertices[730].z) vertices[25] = SCNVector3(x: vertices[25].x, y: vertices[25].y*ll_ratio_y, z: vertices[25].z) vertices[709] = SCNVector3(x: vertices[709].x, y: vertices[709].y*ll_ratio_y, z: vertices[709].z) vertices[725] = SCNVector3(x: vertices[725].x, y: vertices[725].y*ll_ratio_y, z: vertices[725].z) vertices[710] = SCNVector3(x: vertices[710].x, y: vertices[710].y*ll_ratio_y, z: vertices[710].z) let vertexData = Data(bytes: vertices, count: vertices.count * MemoryLayout<SCNVector3>.size) let vertexSource = SCNGeometrySource(data: vertexData, semantic: .vertex, vectorCount: vertices.count, usesFloatComponents: true, componentsPerVector: 3, bytesPerComponent: MemoryLayout<Float>.size, dataOffset: 0, dataStride: MemoryLayout<SCNVector3>.stride) let indices: [UInt16] = faceGeometry.triangleIndices.map(UInt16.init) let indexData = Data(bytes: indices, count: indices.count * MemoryLayout<UInt16>.size) let element = SCNGeometryElement(data: indexData, primitiveType: .triangles, primitiveCount: indices.count / 3, bytesPerIndex: MemoryLayout<UInt16>.size) return SCNGeometry(sources: [vertexSource], elements: [element]) } }
Posted
by akash-ar.
Last updated
.
Post not yet marked as solved
0 Replies
189 Views
Hi there, I've met a problem that in my working project build settings. There is no Metal Compiler Build Setting, but it works well in my demo project. And I'm certain I've move the shader files(.metal) into the bundle. How could I resolve this problem? XCode Version 15.0 13.6.1 (22G313)
Posted Last updated
.
Post not yet marked as solved
0 Replies
185 Views
Is there a way to render stereoscopic (left/right) images in a 2d plane that resides in a swiftUI view? I know this is possible in realityKit shaders, and in immersive metal composits, but is it possible via swiftUI shaders, CAMetalLayer, etc? I'd like to draw a 2d window with standard UI chrome (resize, move etc) that displays stereoscopic content on the flat plane of the window.
Posted Last updated
.