So today we needed to back out of some changes that caused some breakages… or so we thought.
Mainly because https://bugzilla.mozilla.org/show_bug.cgi?id=1142132 had broken bluetooth.
Usually the OTA is turned off once a smoke test blocker like this has been found. Today, it was already turned off for different reasons.
Once the regression range was found, and a specific code is found to be an issue… usually we ask the developer to back out or fix before the 4 pm build that day. Today this particular bug happen to be from a European dev who was already gone for the day.
I had thought maybe it might be as easy as backing out : https://hg.mozilla.org/mozilla-central/rev/d645da56de29 and asked for help from a dev.
The backout itself was clean.
hg backout d645da56de29
hg qnew -f bt-backout.patch
[Basically having a B2G repo and removing the gecko and replacing it with a hg clone of Mozilla Central is what I have on one of my boxes; explained in an earlier blog post. In order to be able to do qnew, I found I have to enable various options by changing my .hgrc located in my user account folder: ~ ; see http://mercurial.selenic.com/wiki/UsingExtensions ]
It turned out that it caused an issue with emulator even though it seemed to be fine on device. We found this out through releng and treeherder:
dholbert had looked into it some and commented in https://bugzilla.mozilla.org/show_bug.cgi?id=1137151#c37 that https://hg.mozilla.org/mozilla-central/rev/d5e5fcb29452 also has to be backed out.
At the same time way that it was coded could have been masking the real issue which may be https://bugzilla.mozilla.org/show_bug.cgi?id=1142364
So the lessons learned here is :
1) it may be safer in the hands of dev to back stuff out
2) even if it does end up falling on QA end, we should push to try as well as test on device as an additional precaution so that we don’t break anything.
3) To note, that making a build for device can pass even though the emulator could fail. Fabrice and Bhavana had looked into how could we build if treeherder was failing and realized there was a difference on building emulator vs device; hence QA did not run into these issues and was able to build successfully.
As an exercise though, I backed out of the other part as well, locally.