After Google gave up on Sceneform, this developer built a better way to bring AR faces to life

When Sceneform and ARCORE were discontinued in 2020, it made creating ARCORE apps on Android more difficult. Inspired by native scene APIs and based on arcore samples, I’ve created AugmentedFaceFragment and AugmentedFaceListener An interface to be able to easily create arcore upperfaces on Android.

I will be releasing 3 articles building different upderfaces features using AugmentedFaceFragment and AugmentedFaceListener Interface.

What are you going to build?

Part 1 of the series will cover the basics AugmentedFaceFragment and AugmentedFaceListener As well as an overview of all helper classes. At the end of this article, you will build a simple demo by writing a few lines of code.

Project preparation

Download the Starter project

Repository clone:

git clone 

What is our starting point?

Our starting point is a modified version of the Arcore SDK sample for augmented faces. The code has been modified in such a way that we can now easily add different textures and objects to the face object.

Project structure

• Helpers – Native Java files from code
• Rendering – Native Java files from sample code handling rendering of AR objects and background

Additional files for the repository:

• AugmentedFaceFragment.kt – Enhanced face display and tracking handles
• AugmentedFaceListener.kt – Add and update handles and reinforced faces
• AugmentedFaceNode.kt – Handles facial rendering including texture and associated 3D models
• AugmentedFaceRenderer.kt – Provide facial texture
• FaceRegion.kt – Provides a 3D model in a specific Facelandmark

Setting up the main activity

<LinearLayout xmlns:android="
    xmlns:tools="
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <androidx.fragment.app.FragmentContainerView android:name="com.ar.arfaces.arface.AugmentedFaceFragment"
        android:id="@+id/face_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_gravity="top" />

</LinearLayout>

class MainActivity : AppCompatActivity(), AugmentedFaceListener {

    private lateinit var binding: ActivityMainBinding

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        binding.faceView.getFragment<AugmentedFaceFragment>().setAugmentedFaceListener(this)
        setContentView(binding.root)
    }

    override fun onFaceAdded(face: AugmentedFaceNode) {}

    override fun onFaceUpdate(face: AugmentedFaceNode) {}

}

in activity_main.xml Clean AugmentedFaceFragment As a key insight and to be able to receive events from this fragment, we need to define and set AugmentedFaceListener To our part.

onFaceAdded The method will be called when Arcore detects a new face and onFaceUpdate It will be called on every frame update.

Let’s add the face texture as the next step.

Add facial texture

We will use the Arcore sample assets for the first face mask. You can find assets/models folder in your project. Let’s add freckles.png As facial texture:

in MainActivity.kt Modify onFaceAdded The method is as follows:

override fun onFaceAdded(face: AugmentedFaceNode) {
   face.setFaceMeshTexture("models/freckles.png")
}

Add 3D models

Introduction to uppertedfacenode

It uses the booster class with the face network and the median to identify facial regions. These areas are:

• Left front (Left_forehead)
• Right front (right – forehead)
• Node Tip (NOSE_TIP)

This project includes AugmentedFaceNode season. Inspired by the scene, it is a node that will provide visual effects on a face detected using ARCORE. AugmentedFaceNode Defines the same facial areas as AugmentedFace In a companion object:

companion object {
   enum class FaceLandmark {
       FOREHEAD_RIGHT,
       FOREHEAD_LEFT,
       NOSE_TIP
   }
}

Later in this tutorial, we will expand FaceLandmark enum class And add areas of our face.

AugmentedFaceNode Includes facial marks HashMapWhich connects FaceLandmark with a 3D model.

The source code for this project can be found here.