Test Automation - Common Problems

測試自動化, 相對於手動測試, 就是寫程式去測試程式, 測試程式由機器運行.

wiki測試自動化的簡介

一旦測試程式寫好後, 可以一直反覆利用.

舉例來說, 每天產生的daily build都跑一次自動測試來確保產品的品質, 

CI (Continuous Integration)產生的build亦可以由此確保品質.

只要有機器, 任何時間任何地點都可以執行, 也不會有人為的疏失.

測試工程師只要看報告就能掌握每天產品狀況.

聽起來好處很多, 不是嗎?

但是為什麼在大部分的公司, 還是以手動測試居多呢?

原因有很多: 最主要是因為

1. 成本高昂 

    根據之前讀到的文章, 測試自動化至少需要被執行 17次以上才符合投資報酬率.

• 資源有限的情況下, 有些測試項用手動測試反而比較符合效率.
• 開發過程中, 程式及介面不斷的改變, 測試程式也需要很多心力去維護.
• 商業用的測試軟體通常要價不斐且較無彈性 V.S. 自己開發測試架構亦需要成本. 

2. 門檻高 

    測試自動化如果不夠穩定, 品質不好, 無法反映出產品真實的品質,其實跟沒做差不多.

• 測試自動化回報錯誤, 但是其實是測試程式本身的錯誤.
• 測試程式不夠穩定導致很多的timing issue, 每次執行的結果不盡相同. 在無法相信測試的狀況下, 最後只能依靠手動測試.

    倘若作的方向不對, 能重複使用的機會也很低.

• 全部測試程式都採用GUI上的play&record. 一旦UI變動或者是開發新版本(UI通常也會改變), 相關的程式都需要修改.
• 測試的方向與架構需要有經驗的人來帶領, 但目前專精測試自動化的人並不多.

除此之外, 也有看到由於第一次經驗不足導致失敗. 便沒有信心在做下去的例子.

因此, 可以想到的是測試自動化絕對沒有想像中容易, 更不是所謂的銀彈.

是一種長期的投資, (比較像推重物, 必須先克服最大靜摩擦力, 方能推動物體.)

但若能突破, 將能把軟體開發往上提升一個境界.

因為在新時代的Scrum, CI等產品開發流程, 測試自動化都是不可或缺的一大前提.

之後將會有主題來討論如何解決這些問題.

----------------------------------------------------------------------------------------------------------------------------------

Automation test, relative to manual test, is a process to write code to test software, which is executed by machines.

Wiki - test automation 

Once test code is finished, it can be used repeatedly to verify software.

For example, daily build can be verified by test automation without manual effort any more.

We can also make sure CI (Continuous Integration) build quality by the same test code.

It can be executed any time any where by machines, and there is no human error.

Test engineers just only need to read testing report to know the quality. 

It sounds great, doesn't it?

But why most companies in industry still rely on manual test?

There are some reasons:

1. Cost is expensive

According to research, test automation needs to be executed at least 17 times to cover the ROI (return on investment).

• Under limited resource, some test cases are executed more efficiently by manual test.
• During software development, program and graphic user interface is usually changed rapidly. It takes lots effort to maintain test code.
• Commercial test tool is usually expensive and less flexible V.S. homemade framework is in need of great effort, too.

2. Entry barrier is high

If test code is unstable, or quality is not good. It can't reflect the true status of software. It may even worse than no test automation.

• Test automation reports bugs, but they actually are test code's bugs.
• There are timing issues in run time, so every time testing result is different. If testing result is not reliable, the only way is verified by manual test.

    If direction of test automation is wrong, test code reuse rate will be low.

• All test code bases on GUI record & replay. Once GUI is changed or next release starts (GUI is possibly changed), related test code needs to be rerecorded again.
• It needs experts to lead direction and architecture of test automation. However, experts are not many in this area.

Besides, some teams may fail in their first try in test automaton, and have no confidence to have second try.
Therefore, we know test automation is not easy to be achieved, and of course is not sliver bullet.
It is a long term investment (Similar to pushing a heavy box, you need to overcome static friction force and then you can push it successfully), and It is essential to success in new era's software developing processes, ex. scrum, CI.
There will be topics to discuss how to resolve these problems.