So, somewhere in this thread I saw a few comments about the count of Google tests.
Assuming I have a good knowledge on these based on my past experiences, I’d like to give you more clear picture on them.
Let’s just use active Fairphone devices for now.
So its,
FP2 - running Android 10
FP3 - running Android 11
FP4 - running Android 11
But, FP2 is a 32bit device where as FP3 and FP4 are 64bit devices
Since FP2 is a 32 bit device with a Lollipop launch, it does not support VTS(Vendor Test Suite) or CTS-on-GSI but since both FP3 and FP4 are post Pie launch, they have these two suites in addition to CTS + GTS(GMS Test Suite) + STS(Security Test Suite).
FP2
CTS - 477000 approx + GTS - 2600 approx + STS - 900 approx
Now let’s come to Android 11 64 bit devices
CTS - 1659750 approx + GTS - 3900 approx + STS - 950 approx + VTS - 119900 approx + CTS-on-GSI - 152400 approx
Now, the same devices if I have to give an overview on Android 12!(Note: There’s no reference for this and these numbers are just an estimation)
CTS - 2292200 approx + GTS - 3900 approx + STS - 980 approx + VTS - 114600 approx + CTS-on-GSI - 165200 approx
Then how will it be for Android 13??
A very high level CTS run on FP3 is showing me the count as 2673000 approx but I shall get the exact count later.
So yes, the Google certification process is a very complicated step in a dessert release for Android. Even a single failure out of these millions of tests will be sufficient for Google to reject the build from getting certified.
Besides this, as someone already touched upon, the operator approval. They too would be having some requirements that has to be satisfied before they accept the build and make it available to their users.
On top of this would be the user visible/ user facing issues that would be addressed from time to time.
I hope I have given you all a brief overview on the process. Do let me know if you have any queries and I would be happy to answer them.
Thanks.
TeamB58.