Life after Death by PowerPoint

我最愛的影片之一.

Don McMillan 告訴你千萬別這樣做投影片.

One of my favorite videos.

Don McMillan shows examples of how NOT to do Powerpoint.

Key to Successful Automation I - Target

在一間小會議室, 二個工程師正在討論下一專案的自動化測試.
"Julian, 這次的測試自動化要怎麼做呢?" 我同事問.
"應該從前一版最花時間的測試先下手吧." 我回答.
"那應該是回歸測試(Regression Test). 這對我們是一個挑戰, 因為這次的測試工程師比上一版本時更少." 他說道.
"恩, 我們可以先將驗收測試(Acceptance Test)自動化, Acceptance test在專案中會被執行數百次." 我說道.
"自動化測試的重複使用也很重要, 我們不可能每次都砍掉重寫. 如果我是老闆, 一定不願意投資不能重複使用的自動化測試." 我馬上接著說.
"我同意, 不過這並不容易. 因為使用者介面不斷地改變, 何況我們也不清楚下一版本專案的規格, 甚至整個專案架構都會有所改變. 有太多不確定因素了." 他說道.
"你的考量很對, 但也只能先試著用對的方法去做. 一般狀況下, 越過使用者介面的自動化測試比較有機會能重複利用, 因為程式動到底層的機會比較小." 我說道.
"好, 我們應該要選擇最有價值的方法." 他同意我的說法.


你可能已經瞭解自動化測試帶來的好處, 也有決心去做好它.
但是通常會遇到的第一個問題是"要怎麼開始?"

在開始研究自動化測試的框架(framework)及可用的程式語言之前.
更重要的是瞭解測試自動化的目標.
測試自動化只是測試方法的一種, 它絕不是銀彈能拿來解決任何問題. 它有些限制: 成本昂貴及入門門檻高, 也不適合能來做某些類型的測試.
比較好的做法是, 根據之前專案的經驗, 找出最需要測試自動化的部分, 並訂定目標去解決問題.
舉例來說, 在出hotfix之前, 通常都要花費很多精力測試有沒有改出新的bug.
這個例子中, 可以將目標設定為將基本功能的測試都自動化, 這樣每次新的hotfix釋出之前, 只要跑過自動化測試就能比較安心地將hotfix給用戶.
還有一些常看到的目標: 測試平台組合太多, 需要自動化測試幫忙. 或是需要一套關鍵字測試自動化系統來節省測試工程師開發的時間. 或是高重複使用率的測試自動化.

目標決定好之後, 測試自動化的方向就不容易走偏. 可以開始尋找或是開發框架及工具, 向有同樣經驗的人諮詢, 並可以規畫整個測試自動化的策略. "目標不一定是需要達到的, 目標往往只是用來說明瞄準的方向." 李小龍說過.

第一次的測試自動化是很艱難的. 即使以失敗告終, 也不要灰心. 在過程中可以學習到很多, 而且方向對的話, 成果只會一次比一次好.
自動化測試是很值得嘗試的. 它不只能讓測試工程師多一種測試的選擇, 也能帶來一定的成就感. 當我們有更多種選擇的時候, 也代表我們能用更有效率的方法來測試. 在這個敏捷開發(Agile Development)的時代中, 測試時間永遠都是不夠, 我們應該要能選擇最適合的方法來測試軟體.

你準備好開始自動化了嗎?

--------------------------------------------------------------------------------------------------------
In a small meeting room, two engineers were discussing test automation for next project.
"Julian, what is our automation plan?" My colleague asked.
"Automate the most effort consuming work in previous project." I answered.
"I believe it is regression test effort. It is a challenge for us. In this project, there are even less test engineers than last one." He said.
"Yes, we should focus on automating acceptance test first. Acceptance test is ususally run a hundred times during development" I said.
"I suggest test automation reuse rate be considered, too, because we shouldn't redevelop test automation in every project, and then drop it. If I were boss, I don't want to invest automation which can't be reused in next version." I also said.
"I agree, but it is not easy, because user interface is always changing, and nobody knows the spec of next version now; maybe there will be code re-architecture. There are too many uncertainties." He said.
"Your concern is right. However, in general cases, automation bypassing user interface can be reused because the layer behind user interface is rarely changed, we should try to make it." I said.
"OK, we should select the most valuable way as our direction." He agreed with me.


You may understand the benefits of test automation, and have determination to adopt it.
The next problem is :"How to start it?"

Before starting to survey testing framework or scripting language for test automation.
The more important thing is to understand what is the target of automation.
Automation is one of test methodologies, however, it is not a silver bullet. Automation cost is expensive and entry point is high, and  it is not suitable for some kind of testing.
Suggestion is to refer to the point of suffering in current testing, and set target to solve it.
For example, in the project of sustained product, it takes huge efforts to test side effects of hotfix, because there might be side effects of basic functions after applying hotfix.
In this case, we can set the target to automate basic function testing to reduce regression efforts for every hotfix.
There are other kinds of targets, including the target of executing tests on comprehensive testing platforms, or the target of creating key-word based test automation library for other testing engineers to make up test scripts easily, or the target of high reuse rate test automation.

Once you decide the target, it helps you find correct direction for automation.
Based on it, you can survey or develop suitable tools and testing framework, consult with experienced people, and work on automation strategy. "A goal is not always meant to be reached, it often serves simply as something to aim at." Said by Bruce Lee

The first automation is not easy. If it fails in the end, don't be dejected. There are lots of things to be learned, and it will get better and better if the target is correct.
Automation is still worthy of investment. It can complete test engineers' skill set, and it can inspire test engineers by the sense of achievement. The more skills we have, the more efficient we are. Testing time is never enough in the era of Agile development, so, we need to select the best way to test software.

Are you ready to start automation?

What do you do? Checking v.s.Testing

~三年前我的一天~
10:00 AM
一如往常, 早上十點左右進辦公室,
打開電腦, 看郵件. "很好, 今天的daily build沒有build fail"
將我的虛擬機器 (Windows XP x86) 還原.
開始安裝最新的build, 等待的過程中繼續看信.
過了十分鐘左右, 安裝成功, 開始執行防火牆的基本功能測試.

11:00 AM
由於基本功能測試已經跑了幾十次, 我早已駕輕就熟.
"XP平台上的基本功能運作良好" 我心裡想
但由於產品的驅動程式在不同平台上都不同.
保險起見,
還必須測試基本功能在另外2台虛擬機器上(Windows Vista x86, Windows Vista x64).

12:00 AM
測試結束, 3個平台的基本功能皆正常.
但對於每個build的例行公事感到無奈甚至厭煩.
與同事一同去吃中餐.

~Checking v.s.Testing~
在產品開發過程中, 為了確保每天build的品質.
通常都會有所謂的acceptance test.
唯有這些基本測試項目通過後, 我們才能確定這個build是有品質的.
但由於測試項目固定.
執行數十次後, 覺得自己就像是個機器人.
不, 事實上可能比機器人還不如, 因為機器人還不會犯錯.
如此一來很可惜地浪費了人的聰明才智, 更有可能賠上的是對測試的熱情.

前陣子看到Michael Bolton的文章在探討Testing 跟Checking, 還蠻有趣的.
Checking Is Confirmation. 

"Checking是一種確定的過程"

我們已經對行為有所預期, 只是去檢查是否合乎預期.

舉個例子來說, 就像執行已經定義好的測試項: 點擊桌面的捷徑後軟體介面應該被開啟.
Testing Is Exploration and Learning.

"Testing是一種探索與學習的過程"

與軟體有所互動.

不斷地觀察, 思考, 探索, 學習, 

舉例來說, 觀察軟體在千禧年下的反應, 

從結果再去推想是否其他部分也有受到日期的影響, 

文章中有提到一個點: Checks Are Machine-Decidable.

當預期結果是可以描述的時候, 機器或是其他人已經有能力和我們作一樣的檢查.

既然如此, 

大家可以思考看看, 每天的工作中有多少是所謂的Checking?
是不是有機會自動化Checking的部分呢?
讓自己從無盡的測試中鬆一口氣, 去做更有價值的事.

----------------------------------------------------------------------------------------------------------------------------------

~One day three years ago~

10:00 AM

As usual, I arrived to office around 10 o'clock.
Powered on working machine, and checked emails. "Great, there is no failed build today"
Reverted my virtual machine (Windows XP x86) to clean snapshot.

I installed the latest build on virtual machine. During installation, I checked other emails.
After 10 minutes, installation was successful, and I began to test basic functions of firewall.

11:00 AM
Because I had executed test cases dozens of times, I was very familiar with them.
"Basic functions work well on Windows XP platform" I thought.
However, drivers of software were different in every platform.
For safety, I had to test basic functions on other 2 virtual machines (Windows Vista x86, Windows Vista x64)

12:00 AM
Tests were finished, all basic functions worked well on 3 platforms.
I was bored by these routine checks of every build.

Had launch with my colleagues.

~Checking v.s.Testing~

In software development, there is build acceptance test to make sure quality of daily build.

Only if all basic test cases pass, we can say this build is qualified.

However, these basic test cases are fixed.

After dozens of times of execution, people may feel what they act is like a robot.

No, maybe worse than a robot because people may make errors, but it doesn't.

It is a pity to waste wisdom of humans, and lose passion of testing.

I read article from Michael Bolton to discuss about Testing and Checking recently.

It is very interesting.

"Checking Is Confirmation." He said.

We already have expected result, and we just confirm if it meets our expectation.

For example, a predefined test case: Main console should be launched if user clicks shortcut.

"Testing Is Exploration and Learning." He also said.

We interact on software.

Observe, think, explore, and learn it.

For example, we observe the interaction of millennium.

Then we may be inspired and think about if other components are effect by date, too.

There are another interesting words in the article. "Checks Are Machine-Decidable."

If expected result is describable, machines or other people should already have abilities to check it.

So you can think about it, how much of your daily work is checking?

Is it possible to automate it?

Rescue yourself from endless routine checking, and do something more valuable.

Test Automation - Common Problems

測試自動化, 相對於手動測試, 就是寫程式去測試程式, 測試程式由機器運行.

wiki測試自動化的簡介

一旦測試程式寫好後, 可以一直反覆利用.

舉例來說, 每天產生的daily build都跑一次自動測試來確保產品的品質, 

CI (Continuous Integration)產生的build亦可以由此確保品質.

只要有機器, 任何時間任何地點都可以執行, 也不會有人為的疏失.

測試工程師只要看報告就能掌握每天產品狀況.

聽起來好處很多, 不是嗎?

但是為什麼在大部分的公司, 還是以手動測試居多呢?

原因有很多: 最主要是因為

1. 成本高昂 

    根據之前讀到的文章, 測試自動化至少需要被執行 17次以上才符合投資報酬率.

• 資源有限的情況下, 有些測試項用手動測試反而比較符合效率.
• 開發過程中, 程式及介面不斷的改變, 測試程式也需要很多心力去維護.
• 商業用的測試軟體通常要價不斐且較無彈性 V.S. 自己開發測試架構亦需要成本. 

2. 門檻高 

    測試自動化如果不夠穩定, 品質不好, 無法反映出產品真實的品質,其實跟沒做差不多.

• 測試自動化回報錯誤, 但是其實是測試程式本身的錯誤.
• 測試程式不夠穩定導致很多的timing issue, 每次執行的結果不盡相同. 在無法相信測試的狀況下, 最後只能依靠手動測試.

    倘若作的方向不對, 能重複使用的機會也很低.

• 全部測試程式都採用GUI上的play&record. 一旦UI變動或者是開發新版本(UI通常也會改變), 相關的程式都需要修改.
• 測試的方向與架構需要有經驗的人來帶領, 但目前專精測試自動化的人並不多.

除此之外, 也有看到由於第一次經驗不足導致失敗. 便沒有信心在做下去的例子.

因此, 可以想到的是測試自動化絕對沒有想像中容易, 更不是所謂的銀彈.

是一種長期的投資, (比較像推重物, 必須先克服最大靜摩擦力, 方能推動物體.)

但若能突破, 將能把軟體開發往上提升一個境界.

因為在新時代的Scrum, CI等產品開發流程, 測試自動化都是不可或缺的一大前提.

之後將會有主題來討論如何解決這些問題.

----------------------------------------------------------------------------------------------------------------------------------

Automation test, relative to manual test, is a process to write code to test software, which is executed by machines.

Wiki - test automation 

Once test code is finished, it can be used repeatedly to verify software.

For example, daily build can be verified by test automation without manual effort any more.

We can also make sure CI (Continuous Integration) build quality by the same test code.

It can be executed any time any where by machines, and there is no human error.

Test engineers just only need to read testing report to know the quality. 

It sounds great, doesn't it?

But why most companies in industry still rely on manual test?

There are some reasons:

1. Cost is expensive

According to research, test automation needs to be executed at least 17 times to cover the ROI (return on investment).

• Under limited resource, some test cases are executed more efficiently by manual test.
• During software development, program and graphic user interface is usually changed rapidly. It takes lots effort to maintain test code.
• Commercial test tool is usually expensive and less flexible V.S. homemade framework is in need of great effort, too.

2. Entry barrier is high

If test code is unstable, or quality is not good. It can't reflect the true status of software. It may even worse than no test automation.

• Test automation reports bugs, but they actually are test code's bugs.
• There are timing issues in run time, so every time testing result is different. If testing result is not reliable, the only way is verified by manual test.

    If direction of test automation is wrong, test code reuse rate will be low.

• All test code bases on GUI record & replay. Once GUI is changed or next release starts (GUI is possibly changed), related test code needs to be rerecorded again.
• It needs experts to lead direction and architecture of test automation. However, experts are not many in this area.

Besides, some teams may fail in their first try in test automaton, and have no confidence to have second try.
Therefore, we know test automation is not easy to be achieved, and of course is not sliver bullet.
It is a long term investment (Similar to pushing a heavy box, you need to overcome static friction force and then you can push it successfully), and It is essential to success in new era's software developing processes, ex. scrum, CI.
There will be topics to discuss how to resolve these problems.

Why Test?

4年前, 我剛畢業並進入軟體公司, 陰錯陽差地成為測試工程師.
雖然與一開始的預期不太相同.
但是我發現這件工作也是蠻有趣,
RD的工作在學生時代多或少有接觸,
相對而言, 測試對我來說是新的挑戰.

初期, 我測試的部分是防毒軟體中的防火牆.
理所當然地需要很多網路相關的知識.
還有學習產品相關的知識.
之後, 學習如何去撰寫Test case,
等到有Daily build後, 執行Test case.
一切就跟大家在軟體工程中看到的描述差不多.

過了4年, 有些朋友從測試工程師轉成RD.
就目前的業界發展來看, 除了大公司外是沒有什麼測試工作的職缺, 就算有待遇也是很差.
沒接觸過的朋友們也會問我 "那不是找些工讀生點點介面就可以了嗎"
在一些討論版更是一面倒地勸人不要做測試, 要當RD.
事實上我也曾有機會可以轉RD.但是我並沒有答應還是留在測試這個領域.
有幾個原因可以給大家參考:

1. 我的個性還蠻適合測試的. 這點很重要, 要有堅持並且細心.
2. 測試是到使用者前的最後一關, 好的測試工程師的思緒清楚而且視野寬廣, 我一直期許達能到這個境界 並藉由測試把產品導向更好的一面. 做出一個好產品而且很多人用是很有成就感的.
3. 第二年起的三年間我都是從事自動化測試. 不但有寫到程式, 而且我發現這是個很有趣的領域. 因為RD的程式有人測試, 但是自動化測試的程式碼是沒有太多資源可以幫忙檢驗的. (不然就沒完沒了, 找人測試測試程式). 要如何把自動化測試做好是非常有挑戰性的一件工作.

所以很幸運地, 我避開了測試中最令人感到煩躁的部分 - 無盡地反覆執行一樣的 Test case.

並也算是半個RD - 寫程式在測試程式,

這些原因讓我的熱情在今天還能持續著.

----------------------------------------------------------------------------------------------------------------------------------

4 years ago, I just graduated from school and applied a software engineer job.
I got a job but it was different from what I originally thought - developer.
I accidentally became a software engineer in test.
Soon, I found that this job was also interesting.
Developers' job I had experienced when I was a student.
However, test is a whole new challenge to me.

In the beginning, my testing component was firewall in anti-virus software.
It was essential for me to study knowledge of networks, and knowledge of product itself.
Then, studied how to create test cases, and executed test cases.
What I did was almost the same as description in software book.

Today, some my friends in testing area have transfered to developers.
In current industry, except big software companies, head count of engineers in test is much fewer,  and pay is not very good.
"What is testing? Hire Interns to click UI to check if application crashes" my friend told to me.
People advise freshmen that it is not good to apply test engineers in many famous forums.
Actually I had chances to transfer to developer before, but I didn't.
There are some reasons:

1. Testing is suit with my personality, and it is the most important. Testing requires persistence and circumspection.
2. Testing is the last guard before software releasing to customers. A good test engineer has clear thought and far vision. I always wish I can I achieve that level, and direct software to the better level, too. There is a lot sense of achievement to deliver great softwares to people.
3. From the 2th year, I focused on test automation in 3 years. Not only aligned with my original expectation to write code, but also I found test automation is an interesting area. Developer's code has test engineers to test it, but there is no additional resource to test test code. (It will be infinite loop if finding someone to test test code). How to achieve good test automation is a big challenge.

Fortunately, I avoid the most boring part of test - unlimited regression manual test.

And I act as a semi-developer: writing code to test code.

These reasons have kept my passion until today.