Test Automation - Common Problems

測試自動化, 相對於手動測試, 就是寫程式去測試程式, 測試程式由機器運行.

wiki測試自動化的簡介

一旦測試程式寫好後, 可以一直反覆利用.

舉例來說, 每天產生的daily build都跑一次自動測試來確保產品的品質, 

CI (Continuous Integration)產生的build亦可以由此確保品質.

只要有機器, 任何時間任何地點都可以執行, 也不會有人為的疏失.

測試工程師只要看報告就能掌握每天產品狀況.

聽起來好處很多, 不是嗎?

但是為什麼在大部分的公司, 還是以手動測試居多呢?

原因有很多: 最主要是因為

1. 成本高昂 

    根據之前讀到的文章, 測試自動化至少需要被執行 17次以上才符合投資報酬率.

• 資源有限的情況下, 有些測試項用手動測試反而比較符合效率.
• 開發過程中, 程式及介面不斷的改變, 測試程式也需要很多心力去維護.
• 商業用的測試軟體通常要價不斐且較無彈性 V.S. 自己開發測試架構亦需要成本. 

2. 門檻高 

    測試自動化如果不夠穩定, 品質不好, 無法反映出產品真實的品質,其實跟沒做差不多.

• 測試自動化回報錯誤, 但是其實是測試程式本身的錯誤.
• 測試程式不夠穩定導致很多的timing issue, 每次執行的結果不盡相同. 在無法相信測試的狀況下, 最後只能依靠手動測試.

    倘若作的方向不對, 能重複使用的機會也很低.

• 全部測試程式都採用GUI上的play&record. 一旦UI變動或者是開發新版本(UI通常也會改變), 相關的程式都需要修改.
• 測試的方向與架構需要有經驗的人來帶領, 但目前專精測試自動化的人並不多.

除此之外, 也有看到由於第一次經驗不足導致失敗. 便沒有信心在做下去的例子.

因此, 可以想到的是測試自動化絕對沒有想像中容易, 更不是所謂的銀彈.

是一種長期的投資, (比較像推重物, 必須先克服最大靜摩擦力, 方能推動物體.)

但若能突破, 將能把軟體開發往上提升一個境界.

因為在新時代的Scrum, CI等產品開發流程, 測試自動化都是不可或缺的一大前提.

之後將會有主題來討論如何解決這些問題.

----------------------------------------------------------------------------------------------------------------------------------

Automation test, relative to manual test, is a process to write code to test software, which is executed by machines.

Wiki - test automation 

Once test code is finished, it can be used repeatedly to verify software.

For example, daily build can be verified by test automation without manual effort any more.

We can also make sure CI (Continuous Integration) build quality by the same test code.

It can be executed any time any where by machines, and there is no human error.

Test engineers just only need to read testing report to know the quality. 

It sounds great, doesn't it?

But why most companies in industry still rely on manual test?

There are some reasons:

1. Cost is expensive

According to research, test automation needs to be executed at least 17 times to cover the ROI (return on investment).

• Under limited resource, some test cases are executed more efficiently by manual test.
• During software development, program and graphic user interface is usually changed rapidly. It takes lots effort to maintain test code.
• Commercial test tool is usually expensive and less flexible V.S. homemade framework is in need of great effort, too.

2. Entry barrier is high

If test code is unstable, or quality is not good. It can't reflect the true status of software. It may even worse than no test automation.

• Test automation reports bugs, but they actually are test code's bugs.
• There are timing issues in run time, so every time testing result is different. If testing result is not reliable, the only way is verified by manual test.

    If direction of test automation is wrong, test code reuse rate will be low.

• All test code bases on GUI record & replay. Once GUI is changed or next release starts (GUI is possibly changed), related test code needs to be rerecorded again.
• It needs experts to lead direction and architecture of test automation. However, experts are not many in this area.

Besides, some teams may fail in their first try in test automaton, and have no confidence to have second try.
Therefore, we know test automation is not easy to be achieved, and of course is not sliver bullet.
It is a long term investment (Similar to pushing a heavy box, you need to overcome static friction force and then you can push it successfully), and It is essential to success in new era's software developing processes, ex. scrum, CI.
There will be topics to discuss how to resolve these problems.

Why Test?

4年前, 我剛畢業並進入軟體公司, 陰錯陽差地成為測試工程師.
雖然與一開始的預期不太相同.
但是我發現這件工作也是蠻有趣,
RD的工作在學生時代多或少有接觸,
相對而言, 測試對我來說是新的挑戰.

初期, 我測試的部分是防毒軟體中的防火牆.
理所當然地需要很多網路相關的知識.
還有學習產品相關的知識.
之後, 學習如何去撰寫Test case,
等到有Daily build後, 執行Test case.
一切就跟大家在軟體工程中看到的描述差不多.

過了4年, 有些朋友從測試工程師轉成RD.
就目前的業界發展來看, 除了大公司外是沒有什麼測試工作的職缺, 就算有待遇也是很差.
沒接觸過的朋友們也會問我 "那不是找些工讀生點點介面就可以了嗎"
在一些討論版更是一面倒地勸人不要做測試, 要當RD.
事實上我也曾有機會可以轉RD.但是我並沒有答應還是留在測試這個領域.
有幾個原因可以給大家參考:

1. 我的個性還蠻適合測試的. 這點很重要, 要有堅持並且細心.
2. 測試是到使用者前的最後一關, 好的測試工程師的思緒清楚而且視野寬廣, 我一直期許達能到這個境界 並藉由測試把產品導向更好的一面. 做出一個好產品而且很多人用是很有成就感的.
3. 第二年起的三年間我都是從事自動化測試. 不但有寫到程式, 而且我發現這是個很有趣的領域. 因為RD的程式有人測試, 但是自動化測試的程式碼是沒有太多資源可以幫忙檢驗的. (不然就沒完沒了, 找人測試測試程式). 要如何把自動化測試做好是非常有挑戰性的一件工作.

所以很幸運地, 我避開了測試中最令人感到煩躁的部分 - 無盡地反覆執行一樣的 Test case.

並也算是半個RD - 寫程式在測試程式,

這些原因讓我的熱情在今天還能持續著.

----------------------------------------------------------------------------------------------------------------------------------

4 years ago, I just graduated from school and applied a software engineer job.
I got a job but it was different from what I originally thought - developer.
I accidentally became a software engineer in test.
Soon, I found that this job was also interesting.
Developers' job I had experienced when I was a student.
However, test is a whole new challenge to me.

In the beginning, my testing component was firewall in anti-virus software.
It was essential for me to study knowledge of networks, and knowledge of product itself.
Then, studied how to create test cases, and executed test cases.
What I did was almost the same as description in software book.

Today, some my friends in testing area have transfered to developers.
In current industry, except big software companies, head count of engineers in test is much fewer,  and pay is not very good.
"What is testing? Hire Interns to click UI to check if application crashes" my friend told to me.
People advise freshmen that it is not good to apply test engineers in many famous forums.
Actually I had chances to transfer to developer before, but I didn't.
There are some reasons:

1. Testing is suit with my personality, and it is the most important. Testing requires persistence and circumspection.
2. Testing is the last guard before software releasing to customers. A good test engineer has clear thought and far vision. I always wish I can I achieve that level, and direct software to the better level, too. There is a lot sense of achievement to deliver great softwares to people.
3. From the 2th year, I focused on test automation in 3 years. Not only aligned with my original expectation to write code, but also I found test automation is an interesting area. Developer's code has test engineers to test it, but there is no additional resource to test test code. (It will be infinite loop if finding someone to test test code). How to achieve good test automation is a big challenge.

Fortunately, I avoid the most boring part of test - unlimited regression manual test.

And I act as a semi-developer: writing code to test code.

These reasons have kept my passion until today.

Startup

想弄一個自己的Blog很久了, 終於, 在2010年結束前終於開始了第一步.
Blog的名字: No reservations 是我最愛的節目 - 旅遊生活頻道的 "波登不設限"
希望自己能多學習. 精益求精
會先以software engineering, testing automation, presentation 為三大主題.
將一些之前的心得整理出來.

----------------------------------------------------------------------------------------------------------------------------------
I have wondered having my own blog for a long time.
Finally, this blog is created before end of 2010.
Blog's name comes from my favorite TV program: "A

nthony Bourdain: No Reservations" in Travel Channel.

I hope my leaning is also no reservation, too.

The direction will focus on "software engineering", "testing automation", and "presentation".
The first step is summarizing what I realized in my work.