Post not yet marked as solved
281
Views
Hello everyone,
Could you share the code that Apple's team used there? https://developer.apple.com/videos/play/wwdc2020/10043/
If you have a similar code, where I can check if it can be implemented, but you need to check the models on my device.
I want to create an incredible app, but I don’t have enough experience. Could you share more links/information about that?
Thank you in advance, and I wish you all the best!
Post marked as solved
104
Views
Hi,
does anyone know which type of network Apple uses for classification in the CreateML app? Do they use cnn's?
Any help is appreciated.
Thanks.
Post not yet marked as solved
451
Views
Is the code for the simple jumping jacks, lunges, squats app available to download?
thanks,
Dale
Post not yet marked as solved
263
Views
I am creating an app to detect exercises. I trained the model using create ML. The results in the create ML app are near 100%, but when I am integrating into the application using Vision framework it's always showing only one exercise. That one exercise is "other" where I have trained it for random exercises that I do not have enough clips for its own tag, the confidence score seems to be somewhat random, but very consistent as well. I followed the code exactly from Build an Action Classifier with Create ML for creating ml and requesting VNHumanBodyPoseObservation. Followed this for converting VNHumanBodyPoseObservation to MLMultiArray. (Sports analysis tutorial)
func didOutput(pixelBuffer: CVPixelBuffer) {
		self.extractPoses(pixelBuffer)
}
func extractPoses(_ pixelBuffer: CVPixelBuffer) {
				let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer)
		let request = VNDetectHumanBodyPoseRequest { (request, err) in
				if err == nil {
						if let observations =
								request.results as? [VNRecognizedPointsObservation], observations.count > 0 {
								if let prediction = try? self.makePrediction(observations) {
										print("\(prediction.label), confidence: \(prediction.confidence)")
								}
						}
				}
		}
			do {
				// Perform the body pose-detection request.
				try handler.perform([request])
			} catch {
				print("Unable to perform the request: \(error).\n")
			}
}
func makePrediction(_ observations: [VNRecognizedPointsObservation]) throws -> (label: String, confidence: Double) {
		let fitnessClassifier = try PlayerExcercise(configuration: MLModelConfiguration())
				let numAvailableFrames = observations.count
				let observationsNeeded = 60
				var multiArrayBuffer = [MLMultiArray]()
				for frameIndex in 0 ..< min(numAvailableFrames, observationsNeeded) {
						let pose = observations[frameIndex]
						do {
								let oneFrameMultiArray = try pose.keypointsMultiArray()
								multiArrayBuffer.append(oneFrameMultiArray)
						} catch {
								continue
						}
				}
				
				// If poseWindow does not have enough frames (45) yet, we need to pad 0s
				if numAvailableFrames < observationsNeeded {
						for _ in 0 ..< (observationsNeeded - numAvailableFrames) {
								do {
										let oneFrameMultiArray = try MLMultiArray(shape: [1, 3, 18], dataType: .double)
										try resetMultiArray(oneFrameMultiArray)
										multiArrayBuffer.append(oneFrameMultiArray)
								} catch {
										continue
								}
						}
				}
		let modelInput = MLMultiArray(concatenating: [MLMultiArray](multiArrayBuffer), axis: 0, dataType: .float)
//
//
let predictions = try fitnessClassifier.prediction(poses: modelInput)
return (label: predictions.label, confidence: predictions.labelProbabilities[predictions.label]!)
}
func resetMultiArray(_ predictionWindow: MLMultiArray, with value: Double = 0.0) throws {
		let pointer = try UnsafeMutableBufferPointer<Double>(predictionWindow)
		pointer.initialize(repeating: value)
}
code-block
Even though this feature set indicates 14.0. I had slightly better results when updating to 14.3, even though they are still completely un-usable. On some devices, versions older than 14.3 could not even produce any predictions .
Thank you so much for reading my post, this has been frustrating me for many days!
Post not yet marked as solved
125
Views
Hi!
I check a lot of tutorials on Create ML Action Classification and faced with only examples for single person. My question is it possible to implement multiple person action classification? And if yes where can I find examples of implementation? For example I want to classify 5+ persons that can do jumps, squats, push ups, etc.
Post not yet marked as solved
413
Views
In "Build an Action Classifier with Create ML" a fitness trainer sample project was shown that shows how to go about using action classification with the new Vision API to predict what exercises are performed.
I'd love to play around with the project to learn more about how to implement those or similar features.
Is the sample project available for download somewhere? I'm only finding the "Building a Feature-Rich App for Sports Analysis" sample project for the Vision API.
Thanks a lot!
Post marked as solved
213
Views
I'm looking for an idea how to accurately count action repetitions. Average duration of window size let say is 2 sec. Actions length may vary from 1 to 4 sec. So the window may include from 0.5 to 2 repetitions. Hence question, shall I train different models using video clips with a single action with minimal duration for similar groups of actions? Or maybe if model is trained with single action video clips, can I use a confidence param to count one repetition? Any advice will be very appreciated.
Thank you
Post not yet marked as solved
172
Views
In addition to knowing what type of action the user is performing .. Is there a way to count the repetitions of the exercise being done?
Post not yet marked as solved
170
Views
What is correct format of time or duration for start and end columns in annotation file (csv or json)? I'm trying to build an action classifier using csv annotation. It works in general, but I see some strange behaviour during training process and ready model itself does not recognise actions correctly. For example, during features extracting, sometimes I see "..has xx frames with no person in view." But I'm sure that annotated video frames include moving person. I noticed, that I get such notification, when video contains empty frames at the beginning or at the end.
I tried different formats, 00:07.350 or 7.350 sec but get exactly the same result in both cases. So either time (duration) format is not correct and Create ML framework ignores it and process an entire video or there is another reason of that. Can you please point me out what I'm doing wrong. Any help would be very appreciated.
Post not yet marked as solved
198
Views
In the example the function „selectMostProminentPerson“ is used. But I can‘t find any example on how to implement a function to make a person selection.
Are there any examples I‘m missing?
Post not yet marked as solved
206
Views
Great work on Vision framework.
Would like to know more on solution to improve prediction accuracy when multiple human appear in the camera. Any way to filter?
Thanks
Post not yet marked as solved
360
Views
I have not seen any evidence of this in the announcements, but I am wondering what improvements are there in ARKit 4 with LiDAR depth support related to body motion tracking.
Does pose prediction or height estimation accuracy improve? What sort of improvements are available for this particular application?
Post not yet marked as solved
194
Views
I want to use CreateML to recognize hand movements, but I don’t see clearly in WWDC20 that it can be done.
Post not yet marked as solved
245
Views
I am trying to improve the performance of drawing the skeleton with body tracking as I am getting noticeable lag even when further than 5 metres away, and with stable iPhone XS camera. The tracking is not close to the showcased performance in WWDC-10043 demo video.
I have also tried using:
let request = VNDetectHumanBodyPoseRequest(completionHandler: )
...
however the results were the same, and I also tried using revision 1 of the algorithm:
let request = VNDetectHumanBodyPoseRequest( )
request.revision = VNDetectHumanBodyPoseRequestRevision1
...
and this didn't help either.
Here's my current code:
/// Extracts poses from a frame.
class Predictor {
func processFrame(_ samplebuffer: CMSampleBuffer) throws -> [VNRecognizedPointsObservation] {
// Perform Vision body pose request
let framePoses = extractPoses(from: samplebuffer)
// Select the most promiment person.
guard let pose = framePoses.first else {
return []
}
return framePoses
}
func extractPoses(from sampleBuffer: CMSampleBuffer) -> [VNRecognizedPointsObservation] {
let requestHandler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer, orientation: .down)
let request = VNDetectHumanBodyPoseRequest()
do {
// Perform the body pose-detection request.
try requestHandler.perform([request])
} catch {
print("Unable to perform the request: \(error).\n")
}
return bodyPoseHandler(request: request, error: nil)
}
func bodyPoseHandler(request: VNRequest, error: Error?) -> [VNRecognizedPointsObservation] {
guard let observations =
request.results as? [VNRecognizedPointsObservation] else {
print("Empty observations.\n\n")
return []
}
return observations
}
}
class CameraViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
let observations = try? predictor.processFrame(sampleBuffer)
observations?.forEach { processObservation($0) }
}
func processObservation(_ observation: VNRecognizedPointsObservation) {
// Retrieve all torso points.
guard let recognizedPoints =
try? observation.recognizedPoints(forGroupKey: .all) else {
return
}
let storedPoints = Dictionary(uniqueKeysWithValues: recognizedPoints.compactMap { (key, point) -> (String, CGPoint)? in
return (key.rawValue, point.location)
})
DispatchQueue.main.sync {
let mappedPoints = Dictionary(uniqueKeysWithValues: recognizedPoints.compactMap { (key, point) -> (String, CGPoint)? in
guard point.confidence > 0.1 else { return nil }
let norm = VNImagePointForNormalizedPoint(point.location,
Int(drawingView.bounds.width),
Int(drawingView.bounds.height))
return (key.rawValue, norm)
})
let time = 1000 * observation.timeRange.start.seconds
// Draw the points onscreen.
DispatchQueue.main.async {
self.drawingView.draw(points: mappedPoints)
}
}
}
}
Thanks in advance, I hope you can help me out! :)
Post not yet marked as solved
190
Views
Hi,
I installed big sur (mac os 11.0) beta version. in my mac mini(2018).here i want to train the new action classification model. but when i open my application i didn't find the createml application.but in wwdc 2020 video tag (10043) you mentioned that that createml is the inbuilt application of big sur OS. i tried finding in app store also unfortunately i didn't get it.can you help me from where to download and and how to install it?
