GPUImageView inside SKScene as SKNode material - Playing transparent video on ARKit - gpuimage

Question

In Project -A- I used GPUImageView to display Video (recorded on greenscreen) with transparency. Using the GPUImageChromaKeyBlendFilter, and so on.
and works Superb.
Another project -B- based on ARKIT shows me in the space a plain with VIDEO and it also works fine using SKVideoNode and AVPlayer.
Now the question is to combine it all together in one :) So in space I want to display Video but with transparency ...
Unfortunately, I can not render a GPUImageView on any SpriteKit element, and then add to SKScene, which is an animated texture for SCNPlane, is it possible at all? Or maybe there is other way to render Video with transparencies with ARKit.?
Thx for any suggestions

Answer 1

I had the same task! See my solution here.
I basically implemented ChromaKeyMaterial from Lësha Turkowski and wrote this code to place and play the video.
import UIKit
import ARKit
class ARTransVC: UIViewController{
#IBOutlet weak var arSceneView: ARSCNView!
let configuration = ARWorldTrackingConfiguration()
private var player: AVPlayer = {
guard let url = Bundle.main.url(forResource: "FY3A4278", withExtension: "mp4") else { fatalError() }
return AVPlayer(url: url)
}()
override func viewDidLoad() {
super.viewDidLoad()
self.arSceneView.debugOptions = [ARSCNDebugOptions.showWorldOrigin, ARSCNDebugOptions.showFeaturePoints]
self.arSceneView.session.run(configuration)
//a delay for ARKit to capture the surroundings
DispatchQueue.main.asyncAfter(deadline: .now() + 3) {
// A SpriteKit scene to contain the SpriteKit video node
let spriteKitScene = SKScene(size: CGSize(width: self.arSceneView.frame.width, height: self.arSceneView.frame.height))
spriteKitScene.scaleMode = .aspectFit
spriteKitScene.backgroundColor = .clear
spriteKitScene.scaleMode = .aspectFit
//Create the SpriteKit video node, containing the video player
let videoSpriteKitNode = SKVideoNode(avPlayer: self.player)
videoSpriteKitNode.position = CGPoint(x: spriteKitScene.size.width / 2.0, y: spriteKitScene.size.height / 2.0)
videoSpriteKitNode.size = spriteKitScene.size
videoSpriteKitNode.yScale = -1.0
videoSpriteKitNode.play()
spriteKitScene.addChild(videoSpriteKitNode)
// To make the video loop
self.player.actionAtItemEnd = .none
NotificationCenter.default.addObserver(
self,
selector: #selector(ARTransVC.playerItemDidReachEnd),
name: NSNotification.Name.AVPlayerItemDidPlayToEndTime,
object: self.player.currentItem)
// Create the SceneKit scene
let scene = SCNScene()
self.arSceneView.scene = scene
//Create a SceneKit plane and add the SpriteKit scene as its material
let background = SCNPlane(width: CGFloat(1), height: CGFloat(1))
background.firstMaterial?.diffuse.contents = spriteKitScene
let chromaKeyMaterial = ChromaKeyMaterial()
chromaKeyMaterial.diffuse.contents = self.player
let backgroundNode = SCNNode(geometry: background)
backgroundNode.geometry?.firstMaterial?.isDoubleSided = true
backgroundNode.geometry!.materials = [chromaKeyMaterial]
backgroundNode.position = SCNVector3(0,0,-2.0)
scene.rootNode.addChildNode(backgroundNode)
//video does not start without delaying the player
//playing the video before just results in [SceneKit] Error: Cannot get pixel buffer (CVPixelBufferRef)
DispatchQueue.main.asyncAfter(deadline: .now() + 1) {
self.player.seek(to:CMTimeMakeWithSeconds(1, 1000))
self.player.play()
}
}
}
#objc func playerItemDidReachEnd(notification: NSNotification) {
if let playerItem: AVPlayerItem = notification.object as? AVPlayerItem {
playerItem.seek(to: kCMTimeZero, completionHandler: nil)
}
}