Analysis of Robot reports from OSM Jenkins¶

Date and time of the report: 2026-02-08 08:45:19.030816

Analysed period: 2026-01-18 to 2026-02-08.

1. Retrieval of all currrent data for aggregate analytics¶

Database update:

Retrieving new jobs...

Getting new builds from: osm-stage_3-merge/master, osm-stage_3-merge/v18.0, osm-stage_3-merge/v17.0, osm-stage_3-merge/v16.0, osm-stage_3-merge/v14.0

DONE

Retrieving from database...	
DONE

2. Aggregated analysis of stability¶

2.1 Restricts data to time window for analysis¶

2.2 Finding sequences of successful builds and Robot reports¶

Two values retrieved from the build are key to determine the state of the jubs and test suites at a given moment:

  • build_result is the outcome of the build, reported by Jenkins. It can be: SUCCESS, FAILURE, UNSTABLE or ABORTED.
  • test_result is the summary of the concerned Robot tests. It can be: FAIL, UNAVAILABLE or PASS.

Based on these two states, 3 types of temporal sequences of success/failure are identified per builds and test suites:

  1. Successful builds/failed builds in a row: grp_build_result.
  2. Successful test reports vs. test reports with fails in a row: grp_test_result.
  3. Clean builds and tests vs. failures (of any kind) in a row: grp_success_fail.

For the identification of these sequences, the following mapping applies:

Type of sequence Relevant state OK sequence contains NOK sequence contains Ignore
grp_build_result build_result SUCCESS or UNSTABLE FAILURE ABORTED
grp_test_result test_result PASS FAIL UNAVAILABLE
grp_success_fail test_result PASS FAIL or UNAVAILABLE N/A

The different groupings of segments are detected and a label is added to each sample...

DONE

Then, the duration of each of the sequences of success/failure is determined and a specific dataframe is built summarizing such sequences, to ease their representation and analysis...

DONE

2.3 Finding sequences of pass/fails per test suite¶

DONE

3. Reports¶

3.1 Aggregated success rate per test step¶

No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

3.2 Overall success of Jenkins builds and Robot tests¶

No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

3.3 Sequences of pass/fails per test suites¶

No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image 
Empty dataframe. Skipping.

3.4 Failing days per test suite¶

No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
Out[58]:
job Master branch Release EIGHTEEN Release FOURTEEN Release SEVENTEEN Release SIXTEEN
days_failing days_passing days_run error_rate days_failing days_passing days_run error_rate days_failing days_passing days_run error_rate days_failing days_passing days_run error_rate days_failing days_passing days_run error_rate
suites
Basic 01-Crud Operations On Vim Targets 0.000000 9.607604 9.607604 0.0 0.000000 19.686725 19.686725 0.0 0.0 0.0 0.0 0.0 0.000000 1.869456 1.869456 0.0 0.0 0.5 0.5 0.0
Basic 05-Instantiation Parameters In Cloud Init 0.000000 9.607604 9.607604 0.0 0.000000 19.686725 19.686725 0.0 0.0 0.0 0.0 0.0 0.000000 1.869456 1.869456 0.0 0.0 0.5 0.5 0.0
Basic 06-Vnf With Charm 0.000000 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
Basic 07-Secure Key Management 0.000000 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
Basic 08-Disable Port Security Network Level 0.000000 9.607604 9.607604 0.0 0.000000 19.686725 19.686725 0.0 0.0 0.0 0.0 0.0 0.000000 1.869456 1.869456 0.0 0.0 0.5 0.5 0.0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sa 08-Vnf With Vnf Indicators Snmp 9.607604 0.000000 9.607604 1.0 19.686725 0.000000 19.686725 1.0 0.0 0.0 0.0 0.0 1.869456 0.000000 1.869456 1.0 0.5 0.0 0.5 1.0
Slice 01-Network Slicing 0.000000 9.607604 9.607604 0.0 0.000000 19.686725 19.686725 0.0 0.0 0.0 0.0 0.0 0.000000 1.869456 1.869456 0.0 0.0 0.5 0.5 0.0
Slice 02-Shared Network Slicing 0.000000 9.607604 9.607604 0.0 0.000000 19.686725 19.686725 0.0 0.0 0.0 0.0 0.0 0.000000 1.869456 1.869456 0.0 0.0 0.5 0.5 0.0
Sol003 01-Vnf-Lifecycle-Management 0.000000 9.607604 9.607604 0.0 0.000000 19.686725 19.686725 0.0 0.0 0.0 0.0 0.0 0.000000 1.869456 1.869456 0.0 0.0 0.5 0.5 0.0
Sol003 02-Dualstack Ip Vnfm 0.000000 9.607604 9.607604 0.0 0.000000 19.686725 19.686725 0.0 0.0 0.0 0.0 0.0 0.000000 1.869456 1.869456 0.0 0.0 0.5 0.5 0.0

62 rows × 20 columns

4. Information about the latest builds of relevant jobs¶

Latest build of each job¶

auto_id job build timestamp duration build_result test_result pass_count fail_count
0 9151 osm-stage_3-merge/master 3647 2026-02-08 00:06:09 14242395 UNSTABLE FAIL 474.0 11.0
1 9152 osm-stage_3-merge/v18.0 175 2026-02-07 22:12:08 12628311 SUCCESS FAIL 481.0 4.0
2 9153 osm-stage_3-merge/v17.0 398 2026-02-07 17:49:07 15767976 UNSTABLE FAIL 468.0 17.0
3 9154 osm-stage_3-merge/v16.0 507 2026-02-08 02:41:09 13817140 UNSTABLE FAIL 461.0 11.0
4 9155 osm-stage_3-merge/v14.0 925 2026-02-07 19:08:08 1204432 FAILURE UNAVAILABLE NaN NaN

Click to see the details of build 3647 of osm-stage_3-merge/master

Click to see the details of build 175 of osm-stage_3-merge/v18.0

Click to see the details of build 398 of osm-stage_3-merge/v17.0

Click to see the details of build 507 of osm-stage_3-merge/v16.0

Click to see the details of build 925 of osm-stage_3-merge/v14.0

Failed test suites per job (if any):¶

osm-stage_3-merge/master: (full report)

auto_id name status starttime endtime pass fail failed_test_name failed_keyword
116 258336 Epa 04-Epa Underlay Sriov FAIL 2026-02-08 01:16:58 2026-02-08 01:19:41 7 2 Ping from Vnf1 to Vnf2 Execute Remote Command Check Rc Return Output
118 258338 Gitops 02-Declarative Cluster Registration Oka... FAIL 2026-02-08 01:20:56 2026-02-08 02:20:37 6 7 Register Cluster For Testsuite Register Cluster
136 258356 Sa 08-Vnf With Vnf Indicators Snmp FAIL 2026-02-08 03:32:50 2026-02-08 03:43:44 5 2 Instantiate Network Service Create Network Service

osm-stage_3-merge/v18.0: (full report)

auto_id name status starttime endtime pass fail failed_test_name failed_keyword
161 258381 Epa 04-Epa Underlay Sriov FAIL 2026-02-07 23:26:21 2026-02-07 23:29:03 7 2 Ping from Vnf1 to Vnf2 Execute Remote Command Check Rc Return Output
181 258401 Sa 08-Vnf With Vnf Indicators Snmp FAIL 2026-02-08 01:12:34 2026-02-08 01:23:27 5 2 Instantiate Network Service Create Network Service

osm-stage_3-merge/v17.0: (full report)

auto_id name status starttime endtime pass fail failed_test_name failed_keyword
206 258426 Epa 04-Epa Underlay Sriov FAIL 2026-02-07 19:28:44 2026-02-07 19:31:27 7 2 Ping from Vnf1 to Vnf2 Execute Remote Command Check Rc Return Output
208 258428 Gitops 02-Declarative Cluster Registration Oka... FAIL 2026-02-07 19:32:43 2026-02-07 20:29:07 3 10 Add OKA Packages For The Testsuite Add OKA Package
217 258437 K8S 04-Openldap Helm FAIL 2026-02-07 20:59:10 2026-02-07 21:03:46 11 2 Execute Upgrade Operation Execute NS K8s Action
220 258440 K8S 14-Helm Upgrade FAIL 2026-02-07 21:11:15 2026-02-07 21:16:06 15 1 Execute First NS Upgrade Using KDU_REUSE Flag ... Execute NS K8s Action
226 258446 Sa 08-Vnf With Vnf Indicators Snmp FAIL 2026-02-07 21:43:01 2026-02-07 21:53:56 5 2 Instantiate Network Service Create Network Service

osm-stage_3-merge/v16.0: (full report)

auto_id name status starttime endtime pass fail failed_test_name failed_keyword
251 258471 Epa 04-Epa Underlay Sriov FAIL 2026-02-08 04:00:49 2026-02-08 04:03:30 7 2 Ping from Vnf1 to Vnf2 Execute Remote Command Check Rc Return Output
267 258487 Sa 01-Vnf With Vim Metrics FAIL 2026-02-08 04:52:43 2026-02-08 05:08:03 6 1 Get VNF VIM-based Metrics Wait Until Keyword Succeeds
268 258488 Sa 02-Vnf With Vim Metrics And Autoscaling FAIL 2026-02-08 05:08:03 2026-02-08 05:41:37 11 4 Get VNF VIM-based Metric Before Auto-scaling Wait Until Keyword Succeeds
269 258489 Sa 07-Alarms From Sa-Related Vnfs FAIL 2026-02-08 05:41:37 2026-02-08 06:02:28 14 2 Get Alarm Metric Wait Until Keyword Succeeds
270 258490 Sa 08-Vnf With Vnf Indicators Snmp FAIL 2026-02-08 06:02:28 2026-02-08 06:13:21 5 2 Instantiate Network Service Create Network Service

osm-stage_3-merge/v14.0: (full report)

auto_id name status starttime endtime pass fail failed_test_name failed_keyword

Details of failed tests into failing test suites (if any):¶

osm-stage_3-merge/master: (full report)

auto_id suite_name test_name keyword_name status starttime endtime
101 6745232 Epa 04-Epa Underlay Sriov Ping from Vnf1 to Vnf2 Execute Remote Command Check Rc Return Output FAIL 2026-02-08 01:18:46 2026-02-08 01:18:55
104 6745235 Epa 04-Epa Underlay Sriov Ping from Vnf2 to Vnf1 Execute Remote Command Check Rc Return Output FAIL 2026-02-08 01:18:55 2026-02-08 01:19:03
114 6745267 Gitops 02-Declarative Cluster Registration Oka... Register Cluster For Testsuite Register Cluster FAIL 2026-02-08 01:28:07 2026-02-08 01:44:25
120 6745273 Gitops 02-Declarative Cluster Registration Oka... Create KSU Namespace For All KSUs Create KSU FAIL 2026-02-08 01:44:27 2026-02-08 01:51:32
123 6745276 Gitops 02-Declarative Cluster Registration Oka... Create KSU1 Without Params Create KSU FAIL 2026-02-08 01:51:32 2026-02-08 01:58:37
126 6745279 Gitops 02-Declarative Cluster Registration Oka... Create KSU2 With Params Create KSU FAIL 2026-02-08 01:58:37 2026-02-08 02:05:42
129 6745282 Gitops 02-Declarative Cluster Registration Oka... Delete KSU2 Delete KSU FAIL 2026-02-08 02:05:42 2026-02-08 02:05:43
130 6745283 Gitops 02-Declarative Cluster Registration Oka... Delete KSU1 Delete KSU FAIL 2026-02-08 02:05:43 2026-02-08 02:05:45
131 6745284 Gitops 02-Declarative Cluster Registration Oka... Delete KSU Namespace Delete KSU FAIL 2026-02-08 02:05:45 2026-02-08 02:11:01
139 6745886 Sa 08-Vnf With Vnf Indicators Snmp Instantiate Network Service Create Network Service FAIL 2026-02-08 03:32:53 2026-02-08 03:34:04
144 6745891 Sa 08-Vnf With Vnf Indicators Snmp Get VNF SNMP Metrics Wait Until Keyword Succeeds FAIL 2026-02-08 03:34:04 2026-02-08 03:43:05

osm-stage_3-merge/v18.0: (full report)

auto_id suite_name test_name keyword_name status starttime endtime
163 6746628 Epa 04-Epa Underlay Sriov Ping from Vnf1 to Vnf2 Execute Remote Command Check Rc Return Output FAIL 2026-02-07 23:28:10 2026-02-07 23:28:18
166 6746631 Epa 04-Epa Underlay Sriov Ping from Vnf2 to Vnf1 Execute Remote Command Check Rc Return Output FAIL 2026-02-07 23:28:18 2026-02-07 23:28:26
173 6747282 Sa 08-Vnf With Vnf Indicators Snmp Instantiate Network Service Create Network Service FAIL 2026-02-08 01:12:37 2026-02-08 01:13:49
178 6747287 Sa 08-Vnf With Vnf Indicators Snmp Get VNF SNMP Metrics Wait Until Keyword Succeeds FAIL 2026-02-08 01:13:49 2026-02-08 01:22:49

osm-stage_3-merge/v17.0: (full report)

auto_id suite_name test_name keyword_name status starttime endtime
197 6748024 Epa 04-Epa Underlay Sriov Ping from Vnf1 to Vnf2 Execute Remote Command Check Rc Return Output FAIL 2026-02-07 19:30:32 2026-02-07 19:30:40
200 6748027 Epa 04-Epa Underlay Sriov Ping from Vnf2 to Vnf1 Execute Remote Command Check Rc Return Output FAIL 2026-02-07 19:30:40 2026-02-07 19:30:48
207 6748056 Gitops 02-Declarative Cluster Registration Oka... Add OKA Packages For The Testsuite Add OKA Package FAIL 2026-02-07 19:32:45 2026-02-07 19:37:47
210 6748059 Gitops 02-Declarative Cluster Registration Oka... Register Cluster For Testsuite Register Cluster FAIL 2026-02-07 19:37:47 2026-02-07 19:54:10
216 6748065 Gitops 02-Declarative Cluster Registration Oka... Create KSU Namespace For All KSUs Create KSU FAIL 2026-02-07 19:54:12 2026-02-07 19:54:13
219 6748068 Gitops 02-Declarative Cluster Registration Oka... Create KSU1 Without Params Create KSU FAIL 2026-02-07 19:54:13 2026-02-07 19:54:15
222 6748071 Gitops 02-Declarative Cluster Registration Oka... Create KSU2 With Params Create KSU FAIL 2026-02-07 19:54:15 2026-02-07 19:54:17
225 6748074 Gitops 02-Declarative Cluster Registration Oka... Delete KSU2 Delete KSU FAIL 2026-02-07 19:54:17 2026-02-07 19:54:18
226 6748075 Gitops 02-Declarative Cluster Registration Oka... Delete KSU1 Delete KSU FAIL 2026-02-07 19:54:18 2026-02-07 19:54:20
227 6748076 Gitops 02-Declarative Cluster Registration Oka... Delete KSU Namespace Delete KSU FAIL 2026-02-07 19:54:20 2026-02-07 19:54:22
228 6748077 Gitops 02-Declarative Cluster Registration Oka... Deregister Cluster For Testsuite Deregister Cluster FAIL 2026-02-07 19:54:22 2026-02-07 20:06:27
230 6748079 Gitops 02-Declarative Cluster Registration Oka... Delete OKA Packages For The Testsuite Delete OKA Package FAIL 2026-02-07 20:06:31 2026-02-07 20:11:32
247 6748412 K8S 04-Openldap Helm Execute Upgrade Operation Execute NS K8s Action FAIL 2026-02-07 21:00:30 2026-02-07 21:01:37
250 6748415 K8S 04-Openldap Helm Check Replicas After Upgrade Operation Get Vnf Kdu Replica Count FAIL 2026-02-07 21:01:37 2026-02-07 21:02:09
273 6748494 K8S 14-Helm Upgrade Execute First NS Upgrade Using KDU_REUSE Flag ... Execute NS K8s Action FAIL 2026-02-07 21:12:34 2026-02-07 21:13:40
300 6748678 Sa 08-Vnf With Vnf Indicators Snmp Instantiate Network Service Create Network Service FAIL 2026-02-07 21:43:04 2026-02-07 21:44:16
305 6748683 Sa 08-Vnf With Vnf Indicators Snmp Get VNF SNMP Metrics Wait Until Keyword Succeeds FAIL 2026-02-07 21:44:16 2026-02-07 21:53:17

osm-stage_3-merge/v16.0: (full report)

auto_id suite_name test_name keyword_name status starttime endtime
324 6749420 Epa 04-Epa Underlay Sriov Ping from Vnf1 to Vnf2 Execute Remote Command Check Rc Return Output FAIL 2026-02-08 04:02:37 2026-02-08 04:02:45
327 6749423 Epa 04-Epa Underlay Sriov Ping from Vnf2 to Vnf1 Execute Remote Command Check Rc Return Output FAIL 2026-02-08 04:02:45 2026-02-08 04:02:53
342 6749967 Sa 01-Vnf With Vim Metrics Get VNF VIM-based Metrics Wait Until Keyword Succeeds FAIL 2026-02-08 04:53:25 2026-02-08 05:07:27
372 6749997 Sa 02-Vnf With Vim Metrics And Autoscaling Get VNF VIM-based Metric Before Auto-scaling Wait Until Keyword Succeeds FAIL 2026-02-08 05:08:51 2026-02-08 05:22:53
377 6750002 Sa 02-Vnf With Vim Metrics And Autoscaling Wait VIM-based Metric To Exceed Threshold Wait Until Keyword Succeeds FAIL 2026-02-08 05:22:55 2026-02-08 05:36:57
380 6750005 Sa 02-Vnf With Vim Metrics And Autoscaling Check VIM-based Metric Exceeds Threshold After... Check VIM-based Metric Exceeds Threshold FAIL 2026-02-08 05:38:57 2026-02-08 05:38:57
411 6750036 Sa 07-Alarms From Sa-Related Vnfs Get Alarm Metric Wait Until Keyword Succeeds FAIL 2026-02-08 05:43:45 2026-02-08 05:57:46
412 6750037 Sa 07-Alarms From Sa-Related Vnfs Check Alarms Were Received Wait Until Keyword Succeeds FAIL 2026-02-08 05:57:46 2026-02-08 06:01:15
421 6750046 Sa 08-Vnf With Vnf Indicators Snmp Instantiate Network Service Create Network Service FAIL 2026-02-08 06:02:32 2026-02-08 06:03:42
426 6750051 Sa 08-Vnf With Vnf Indicators Snmp Get VNF SNMP Metrics Wait Until Keyword Succeeds FAIL 2026-02-08 06:03:42 2026-02-08 06:12:44

osm-stage_3-merge/v14.0: (full report)

auto_id suite_name test_name keyword_name status starttime endtime

Powered by osm-analytics