Finding regression range in B2G

So there’s a lot of tricks to finding regression ranges in general that is cut down by using hg bisect or git bisect.  The essence is doing a bisect of where the good build is and the bad build is until you get down to one revision/pull etc. difference.  Sometimes a clue/educated guess of what the bug is can help make the number of trials lower.

With B2G there’s some other tricks.  QA usually uses MC builds and inbound builds to help lower the regression range.  The reason why is because the builds are already there.

Having said that it sometimes isn’t enough due to many commits in between builds.

That’s where knowing how to make your own builds and using the bisect tools could help.

1) First you will have to setup your own repo.  You could try updating my VM from a previous post.  Or you can setup your own repo following :

https://developer.mozilla.org/en-US/Firefox_OS/Building_and_installing_Firefox_OS

2) you’ll want to make sure you’re working with the right branch.  ie, if you’re trying to find inbound revision, then you want to be pulling from inbound for gecko:

http://hg.mozilla.org/integration/mozilla-inbound/

you may also have to check :

http://hg.mozilla.org/integration/b2g-inbound/

as b2g-inbound is a branch only for b2g check-ins.

For gaia, you should be on master:

https://github.com/mozilla-b2g/gaia

If you working on a previous branch such as 2.1, be sure to use the matching gecko/gaia

Hash revs are based on the branches, so one branch may not have the same hash code for the same checkin as another branch.

Tip: I have a directory for each that I soft link to.  I reconfigure the directory based on the device version that I’m working for and then repoint the soft links.  In most cases you won’t have to reconfigure most of the other stuff.  gecko/gaia are probably the things that will change the most.  Of course, you’ll want to make sure the rest of the repo is reasonably up to date for the version of the device you are working with.

2) once you setup the repo for the device or whatever, the next step is to find the checkin for the bad build and the good build.  If you pull from Mozilla’s FTP site :

https://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central-flame-kk/

you’ll notice a file called sources.xml  You can do a find for the hash code for gaia and gecko.  If you’re working off of the builds in the ftp site, you’ll find that the gecko/gaia hash codes are actually from the git.mozilla.org repo.

To confuse things even more, git.mozilla.org is a git repo of various other repos including hg, git, + more.

Git inbound can be found : http://git.mozilla.org/?p=integration/gecko-dev.git  ( inbound, b2g-inbound )

Git MC can be found : http://git.mozilla.org/?p=releases/gecko.git

And Gaia can be found : http://git.mozilla.org/?p=releases/gaia.git

So if you want to work off the hash code from the builds, you will want to use these repos.

Note : There is a chance that the git.mozilla.org gets out of sync with hg and some of the other repos.  This is why I mentioned it secondary.

3) For gaia, you can find the bisection without having to deal with the rest of the B2G repo.  you can just go into the gaia directory of the b2g and do a git bisect.

ie. :

1. git checkout <gaia revision>
2. git bisect start <first known bad> <last known good>

3. git log  ( to verify which revision is head )

4. make production ( to build ) and try to reproduce the issue.

5. Mark if it’s a good build or bad build : via

a) git bisect bad
– will state the revision is bad
b) git bisect good
– will state the revision is good; move forward in time.
6) it should move to the next revision, double check with git log, and repeat 4 and 5 until it gives the report of which rev is good and which rev is bad.

Gecko is similar in that you can do :

./build.sh gecko

to build the gecko in the b2g repo.

and you can

./flash.sh gecko

to flash the gecko on your device.

Oh I forgot to mention if you want to figure out if the issue is in gecko or gaia, you can swap the gecko and gaia and check.  ie from the bad build take the gecko and flash w/ the “good” gaia and vice versa.  if the bad gecko + good gaia broke then it’s the gecko.  if it’s the good gecko + bad gaia then it’s gaia.

If you’ve figured out if it’s gecko or gaia that broke the issue (you’ll want to use a close regression range for the swap), you will probably want to use the older gaia when checking for gecko regression range, and the latest gecko if you’re checking for gaia regression range.

Once you have the regressing PR, you can back out via : git revert <commit > -m 1

If you run into issues ( ie if there’s a conflict ), then you will have to resolve it via figuring out which lines need to be removed and kept.

hg has similar commands to git.  See http://www.wikivs.com/wiki/Git_vs_Mercurial for a comparative command list.

Some other tips is, don’t clobber unless you have to.  One of the things that’s safe about releng builds is that they clobber all the time.  So doing comparisons makes for certainty.  The one thing you’ll find building is that it takes time.  If you clobber then it could take 2 hrs to build.  clobber is referring to deleting the objects and out folder : rm -rf obj-dir , rm -rf out

Posted in Uncategorized | Leave a comment

Compilation on Mac:

If you run into mkfs.vfat issue compiling B2G on Mac, see:

Bug 1024997 - [Flame][Build] Failed to build on Mac OS X 10.9, mkfs.vfat: command not found

vi ./device/qcom/msm8610/BoardConfig.mk

and

"BOARD_USBIMAGE_PARTITION_SIZE_KB :=  " in BoardConfig.mk
Posted in Uncategorized | Leave a comment

Exploratory Test Coverage

Personally, I find that most people don’t pay attention to the passing of test and don’t correlate test cases to test coverage very well so the result numbers  are pretty ambiguous anyhow with traditional methods of testing.

Almost like saying 3 out of 4 dentist prefer this toothpaste over that.  Where did they find these dentists?  After the company bribed them with money?

What do I mean?  You could have tons of test cases covering UI but not necessarily the business logic on how something is suppose to work.  etc. etc.  Of course, with good testers that won’t happen; they would try to get a broad range of tests.  Unless you dig into the tests it’s hard to get a view of all the tests.  One could say that it’s documented.  How many people actually bother looking at the percent coverage though?  Or question it?  “There’s x many tests, of course there must be good coverage!”  I’ve heard that too line so many times yet then there’s those escaped bugs…  Escaped bugs will happen.  I’m just saying that the assumption that you have 100 % test coverage based on x number of tests is just… wrong.  Even the thought of the more tests you have the closer you are to 100%.  Well, that might be true; x number of tests doesn’t tell you how close you are to that 100%.

And as I stated, before running the same tests over and over doesn’t prove as fruitful as running new tests.

I’m trying to come up with a way to display coverage using a flow diagram as way of saying this is what I’ve been looking at.  I’ve also contemplated merging it with James Bach stuff on session based testing.  He says to take notes.  I’m expanding on the note taking to fast diagrams.

The thing with developing software is … is that it keeps changing and evolving.  Those changes have to be reflected in the tests as well.  Until the software gets to a place of “code freeze/string freeze”, it’s hard to lock down test cases.  You’ll spend more cycles maintaining than testing.  In which case, exploratory testing/session based testing has the advantage.

The problem is that it’s very faith based that the tester will do well and it only works w/ good testers.  (or testers that are trained to exploratory test)…  The comparison is like do you send a novice mushroom hunter out or do you send a professional mushroom hunter out and how much more mushroom do you think each will come back with in a shorter amount of time?

Management of course wants some documentation, and rather than making it, “faith based like a religion”, we want it more “fact based like science” in an easy to read, charts/diagrams.  Hence the UI user-flow diagram concept I’m trying to do.

Posted in Uncategorized | Leave a comment

Remember to clean your *.pyc when things go awry…

nhirata_: well that’s the screwest thing…
[11:09am] nhirata_: screwiest thing…
[11:10am] nhirata_: I think when I copied the python file the reference is somehow moved with it?  I went from working with one directory of gaia to a completely different directory of gaia
[11:11am] nhirata_:  File “/Volumes/Projects/master_gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/browser/test_browser_windows.py”, line 28, in test_browser_menu_windows
[11:11am] nhirata_:     browser.tap_menu_button()
[11:11am] nhirata_:   File “/Volumes/Projects/B2G_Flame/gaia/tests/python/gaia-ui-tests/gaiatest/apps/search/regions/browser.py”, line 45, in tap_menu_button
[11:11am] nhirata_:     self.marionette.find_element(*self._menu_button_locator).tap()
[11:12am] krupa|lunch left the chat room. (Quit: Linkinus – http://linkinus.com)
[11:13am] kraj joined the chat room.
[11:14am] nhirata_: Is there any way to clear whatever is being cached?
[11:44am] nhirata_: hrm.  I guess I am just going to have to work in that directory…
[12:06pm] AutomatedTester: nhirata_: delete the *.pyc files
[12:06pm] nhirata_: oh.
[12:06pm] nhirata_: I feel dumb now
[12:07pm] AutomatedTester: nhirata_: everyone, and I mean everyone, gets hit by this regularly
[12:07pm] AutomatedTester: even people who have been doing python for years
[12:07pm] AutomatedTester: sometimes it gets overwritten, sometimes its just cached and doesnt
[12:10pm] AaronMT: I suspect AutomatedTester has written a book about this.
[12:11pm] AutomatedTester: AaronMT: I have but apparently it’s so common that no publishing house wants it
[12:11pm] AutomatedTester: and I really want it edited before I release it
[12:12pm] nhirata_: lol
[12:12pm] smooney left the chat room. (Client exited)
[12:14pm] nhirata_: hrm.  rm -rf *.pyc doesn’t seem to work from a higher directory
[12:15pm] AutomatedTester: nhirata_: you need to do all the directories
[12:15pm] AutomatedTester: one sec, I have an alias for it
[12:15pm] AaronMT: alias pycclean=’find . -name “*.pyc” | xargs -I {} rm -v “{}”‘
[12:16pm] AutomatedTester: find . -name ‘*.pyc’ -exec rm {} \;
[12:16pm] nhirata_: oooo…
[12:16pm] nhirata_: hotness
[12:26pm] AutomatedTester: see what I mean… it’s so common, why have it published

Posted in Uncategorized | Leave a comment

Einstein Quote for Mozillians

“Out of clutter, find simplicity. From discord find harmony. In the middle of difficulty lies opportunity.” – Albert Einstein

From : http://www.folderarchy.com/albert-einstein/

Posted in Planet | Tagged | Leave a comment

There’s a story about Einstein and how he never remembered his phone number: http://www.folderarchy.com/albert-einstein/

Why remember something where you know you can reference things?  From that article I also gathered:

“Out of clutter, find simplicity. From discord find harmony. In the middle of difficulty lies opportunity.” – Albert Einstein

and

“Only put in enough energy and effort so as to pull out of life what really matters, what you really want. Be organized enough that you can find everything you need or want, when you need or want it. But don’t fret over the small stuff. And have a way to be confident you can tell the small stuff from the big stuff. ”

Anyhow, I’ve decided to start using my blog as a means of writing notes for myself, because I tend to forget things, and I started finding my blog as a good source of reference how to do something.  I also have notes in a wiki:  https://wiki.mozilla.org/B2G/QA/Tips_And_Tricks

I use that as a quick reference and then do a find in page to get to what I need to.

Anyhow, in my learning about python and marionette, I needed to learn how to do somethings and Dave Burns was kind enough to point me to some docs. It taught me a few things.

Context Management is important; in real life and in marionette : http://marionette-client.readthedocs.org/en/latest/basics.html?highlight=javascript#context-management

Docs on Web APIs: https://developer.mozilla.org/en-US/docs/Web/API

And the marionette latest references : http://marionette-client.readthedocs.org/en/latest/reference.html

In order to get contents of an element : “you can execute script to give you the innerHTML of the parent element to give you what the DOM has for it”

Thanks, Dave, for the help!

Posted in Uncategorized | Leave a comment

QA Experimentation on Process and Automation

I’ve started coding some tests and I’m placing my own repo as I may not necessarily want them checked into the main repo.

I’m not sure the best method for this, at some point I plan to have a machine running various tests that I’ve created which we might not necessarily want to have run on a constant basis.

Basically I want to stagger my tests that check various things:

For example:
Test set A would have a context menu check for open a new window and open a new window using that.
Test set B would have a context menu check for open a new private window and open a new private window using that

Then I would alternate running test set A and test set B for faster coverage.  Bob  Moss talked about this way back when he was a director of QA.  I think we can get a broad area of coverage + a variety of tests running if we spread out the tests in several sets.

My plan is to get good enough where I can do exploratory testing + some automation in a similar fashion in the areas that I am covering so I have smoke tests + functional tests automated that I can run on a weekly (if not daily) basis.  When we get to stabilization phase that’s when we can triage which tests we want to push into the rest of the repo.  It’s a bit ambitious, I understand with the amount of maintenance I may have to do. and it also depends on the areas of course.

Basically with this, I’m experimenting on how much automation we can balance and leverage before the maintenance cost becomes too high.  My goal is to try to optimize my own processes and see what works best for me.  If it gets overwhelming I will change up.  I’m not even sure I will be able to completely go with this idea as of yet.

One of the other things I was thinking about… Is there a way to have marionette report to moztrap with the results to an associated test?  Because then we can have individuals using the automation tests that doesn’t report to tree herder or jenkins report test runs….

This is the type of experimentation that I do in order to find bugs faster.  It’s how I figured out exploratory testing finds bugs faster than the traditional regression based testing.  I had empirical data and tried this approach on multiple products in various companies to make sure that it wasn’t because of my expert knowledge in the product.  FileMaker Pro, Powerpoint, and it carried on with Firefox for Android.  More over you can find bugs faster by looking at areas that have just recently changed and talking to dev about the changes implemented.  This is also from studying with my QA mentor when I first started QAing.  With experience you can take a look at the changes and figure out what to test for in order to check if a bug exists or not.  i.e. if a variable is initialized properly or not (which can be tested by creating a new profile and seeing if the app crashes), etc.

My mentor had also ran automation on new areas as well as old areas.  This is where I haven’t done any experimentation to see what is the best balance as of yet in terms of maintenance cost.  I’m not sure how successful my experimentation will be; wish me luck!

The new test can be found here :

https://github.com/nhirata/mozilla_b2g_gaia/commit/a332179d6428bcc350ff5669c0d48a68262ed4a5

note:I edited it so that I post a branch to the gaia repo that I already have upon suggestion by my peers.

Posted in QMO | Leave a comment