Got error 4009 'Cluster Failure' from NDBCLUSTER

+---------+--------------+----------------+--------------+-------------+-------+-------------+----------------+----------------------+--------------+--------------------+--------------+---------------+---------------+---------------+
| node_id | block_number | block_instance | comm_node_id | upper_bound | scans | scan_errors | scan_fragments | scan_fragment_errors | transactions | transaction_errors | read_key_ops | write_key_ops | index_key_ops | key_op_errors |
+---------+--------------+----------------+--------------+-------------+-------+-------------+----------------+----------------------+--------------+--------------------+--------------+---------------+---------------+---------------+
|       2 |          245 |              1 |            1 |          50 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |            1 |          75 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |         112 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |            1 |         168 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |            83 |             0 |             0 |
|       2 |          245 |              1 |            1 |         252 |     0 |           0 |             12 |                    0 |            0 |                  0 |           65 |      35576498 |             0 |       2318505 |
|       2 |          245 |              1 |            1 |         378 |     0 |           0 |             84 |                    0 |            0 |                  0 |          190 |      17662739 |             0 |       1120428 |
|       2 |          245 |              1 |            1 |         567 |     0 |           0 |             19 |                    0 |            0 |                  0 |           10 |        203395 |             0 |         21013 |
|       2 |          245 |              1 |            1 |         850 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         49265 |             0 |          6400 |
|       2 |          245 |              1 |            1 |        1275 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         33733 |             0 |          4618 |
|       2 |          245 |              1 |            1 |        1912 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         13437 |             0 |          2486 |
|       2 |          245 |              1 |            1 |        2868 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |          1589 |             0 |           221 |
|       2 |          245 |              1 |            1 |        4302 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |           327 |             0 |            34 |
|       2 |          245 |              1 |            1 |        6453 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |           149 |             0 |            42 |
|       2 |          245 |              1 |            1 |        9679 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |            11 |             0 |             5 |
|       2 |          245 |              1 |            1 |       14518 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             4 |             0 |             2 |
|       2 |          245 |              1 |            1 |       21777 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |       32665 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |       48997 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             6 |             0 |             5 |
|       2 |          245 |              1 |            1 |       73495 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |            1 |      110242 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |      165363 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |      248044 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |      372066 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |      558099 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |      837148 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |            1 |     1255722 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |            1 |     1883583 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |     2825374 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |     4238061 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |     6357091 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |     9535636 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            1 |    14303454 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |          50 |     0 |           0 |             12 |                    0 |            0 |                  0 |            0 |      23185869 |             0 |       1726451 |
|       2 |          245 |              1 |            2 |          75 |     0 |           0 |              5 |                    0 |            0 |                  0 |            0 |         27899 |             0 |          2859 |
|       2 |          245 |              1 |            2 |         112 |     0 |           0 |             20 |                    0 |            0 |                  0 |            0 |          6473 |             0 |          1252 |
|       2 |          245 |              1 |            2 |         168 |     0 |           0 |             15 |                    0 |            0 |                  0 |            0 |          2309 |             0 |           763 |
|       2 |          245 |              1 |            2 |         252 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |        104408 |             0 |           631 |
|       2 |          245 |              1 |            2 |         378 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |       3363902 |             0 |           701 |
|       2 |          245 |              1 |            2 |         567 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         50154 |             0 |           465 |
|       2 |          245 |              1 |            2 |         850 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         18032 |             0 |           235 |
|       2 |          245 |              1 |            2 |        1275 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         12071 |             0 |            95 |
|       2 |          245 |              1 |            2 |        1912 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |          8596 |             0 |            41 |
|       2 |          245 |              1 |            2 |        2868 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |          2429 |             0 |             9 |
|       2 |          245 |              1 |            2 |        4302 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |           299 |             0 |             0 |
|       2 |          245 |              1 |            2 |        6453 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |            78 |             0 |             4 |
|       2 |          245 |              1 |            2 |        9679 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             2 |             0 |             0 |
|       2 |          245 |              1 |            2 |       14518 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             2 |             0 |             0 |
|       2 |          245 |              1 |            2 |       21777 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             2 |             0 |             0 |
|       2 |          245 |              1 |            2 |       32665 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |            2 |       48997 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             3 |             0 |             0 |
|       2 |          245 |              1 |            2 |       73495 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |      110242 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |      165363 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |      248044 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |      372066 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |      558099 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |            2 |      837148 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |     1255722 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |     1883583 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |     2825374 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |     4238061 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |     6357091 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |     9535636 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |            2 |    14303454 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |          50 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |       9250384 |             0 |             0 |
|       2 |          245 |              1 |           67 |          75 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         11153 |             0 |             0 |
|       2 |          245 |              1 |           67 |         112 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |          2093 |             0 |             0 |
|       2 |          245 |              1 |           67 |         168 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |           585 |             0 |             0 |
|       2 |          245 |              1 |           67 |         252 |     0 |           0 |              0 |                    0 |            0 |                  0 |           65 |      13369706 |             0 |             0 |
|       2 |          245 |              1 |           67 |         378 |     0 |           0 |              0 |                    0 |            0 |                  0 |          190 |       6289611 |             0 |             0 |
|       2 |          245 |              1 |           67 |         567 |     0 |           0 |              0 |                    0 |          200 |                  0 |           10 |         62691 |             0 |             0 |
|       2 |          245 |              1 |           67 |         850 |     0 |           0 |              0 |                    0 |           50 |                  2 |            0 |         14951 |             0 |             0 |
|       2 |          245 |              1 |           67 |        1275 |     1 |           0 |              0 |                    0 |            4 |                  0 |            0 |          7257 |             0 |             0 |
|       2 |          245 |              1 |           67 |        1912 |     4 |           0 |              0 |                    0 |            0 |                  0 |            0 |          3577 |             0 |             0 |
|       2 |          245 |              1 |           67 |        2868 |     7 |           0 |              0 |                    0 |            3 |                  0 |            0 |           650 |             0 |             0 |
|       2 |          245 |              1 |           67 |        4302 |     2 |           0 |              0 |                    0 |            6 |                  0 |            0 |           112 |             0 |             0 |
|       2 |          245 |              1 |           67 |        6453 |     4 |           0 |              0 |                    0 |            4 |                  0 |            0 |            39 |             0 |             0 |
|       2 |          245 |              1 |           67 |        9679 |     0 |           0 |              0 |                    0 |            1 |                  0 |            0 |             4 |             0 |             0 |
|       2 |          245 |              1 |           67 |       14518 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             2 |             0 |             0 |
|       2 |          245 |              1 |           67 |       21777 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |       32665 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |       48997 |     0 |           0 |              0 |                    0 |            1 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |       73495 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |      110242 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |      165363 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |      248044 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |      372066 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |      558099 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |      837148 |     0 |           0 |              0 |                    0 |            1 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |     1255722 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |           67 |     1883583 |     0 |           0 |              0 |                    0 |            1 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |     2825374 |     0 |           0 |              0 |                    0 |          379 |                  3 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |     4238061 |     0 |           0 |              0 |                    0 |           46 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |     6357091 |     0 |           0 |              0 |                    0 |           96 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |     9535636 |     0 |           0 |              0 |                    0 |          997 |                  4 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           67 |    14303454 |     0 |           0 |              0 |                    0 |          212 |                  3 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |          50 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |      13935486 |             0 |       1726451 |
|       2 |          245 |              1 |           68 |          75 |     1 |           0 |              0 |                    0 |            0 |                  0 |            0 |         16746 |             0 |          2859 |
|       2 |          245 |              1 |           68 |         112 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |          4381 |             0 |          1252 |
|       2 |          245 |              1 |           68 |         168 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |          1807 |             0 |           763 |
|       2 |          245 |              1 |           68 |         252 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |      22311200 |             0 |       2319136 |
|       2 |          245 |              1 |           68 |         378 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |      14737030 |             0 |       1121129 |
|       2 |          245 |              1 |           68 |         567 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |        190858 |             0 |         21478 |
|       2 |          245 |              1 |           68 |         850 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         52346 |             0 |          6635 |
|       2 |          245 |              1 |           68 |        1275 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         38547 |             0 |          4713 |
|       2 |          245 |              1 |           68 |        1912 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |         18456 |             0 |          2527 |
|       2 |          245 |              1 |           68 |        2868 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |          3368 |             0 |           230 |
|       2 |          245 |              1 |           68 |        4302 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |           514 |             0 |            34 |
|       2 |          245 |              1 |           68 |        6453 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |           188 |             0 |            46 |
|       2 |          245 |              1 |           68 |        9679 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             9 |             0 |             5 |
|       2 |          245 |              1 |           68 |       14518 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             4 |             0 |             2 |
|       2 |          245 |              1 |           68 |       21777 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             2 |             0 |             0 |
|       2 |          245 |              1 |           68 |       32665 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |           68 |       48997 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             9 |             0 |             5 |
|       2 |          245 |              1 |           68 |       73495 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |           68 |      110242 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |      165363 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |      248044 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |      372066 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |      558099 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |           68 |      837148 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             1 |             0 |             0 |
|       2 |          245 |              1 |           68 |     1255722 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |     1883583 |     0 |           0 |              0 |                    0 |            0 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |     2825374 |     0 |           0 |              0 |                    0 |          303 |                  0 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |     4238061 |     0 |           0 |              0 |                    0 |          424 |                  2 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |     6357091 |     0 |           0 |              0 |                    0 |          360 |                  1 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |     9535636 |     0 |           0 |              0 |                    0 |         1364 |                  1 |            0 |             0 |             0 |             0 |
|       2 |          245 |              1 |           68 |    14303454 |     0 |           0 |              0 |                    0 |          821 |                  2 |            0 |             0 |             0 |             0 |
+---------+--------------+----------------+--------------+-------------+-------+-------------+----------------+----------------------+--------------+--------------------+--------------+---------------+---------------+---------------+

Looks very good, but there is a lot of key_op_errors

One interesting thing of note here is the transaction time and transaction errors.

Many transactions take more than 14 seconds to complete. So this is definitely not normal
behaviour although it in itself doesn’t generate 4009

But the latency comes from outside the data nodes, the actual RonDB operations complete
in the hundreds of micros

It also reports a number of transaction errors

just curious… how does this affect regular key value lookup select statements.

these errors could be because i was previously running a lot of upsert statements to push data from kafka to rondb :thinking:

Yep, could very well be the case.

There is really nothing that indicates that 4009 should happen.
And if it did happen, the cluster log should really report that the
connection was broken. So don’t really see any reason how it
could happen.

One more ndbinfo table you could look at is
select * from transporters;

mysql> select * from transporters;
+---------+----------------+------------+----------------+-------------+----------------+---------------+------------+----------------+----------+----------------+
| node_id | remote_node_id | status     | remote_address | bytes_sent  | bytes_received | connect_count | overloaded | overload_count | slowdown | slowdown_count |
+---------+----------------+------------+----------------+-------------+----------------+---------------+------------+----------------+----------+----------------+
|       1 |             65 | CONNECTED  | xxx.xxx.xxx.xxx  |   283424596 |       75388908 |             1 |          0 |              0 |        0 |              0 |
|       1 |             67 | CONNECTED  | xxx.xxx.xxx.xxx  |    97425132 |        8880404 |             2 |          0 |              0 |        0 |              0 |
|       1 |            231 | CONNECTING | xxx.xxx.xxx.xxx  |       10808 |            312 |             4 |          0 |              0 |        0 |              0 |
|       1 |              2 | CONNECTED  | -                | 59699258304 |    63439655176 |             0 |          0 |              0 |        0 |              0 |
|       1 |              3 | CONNECTING | -                |           0 |              0 |             0 |          0 |              0 |        0 |              0 |
|       1 |             66 | CONNECTING | -                |           0 |              0 |             0 |          0 |              0 |        0 |              0 |
|       1 |             68 | CONNECTED  | xxx.xxx.xxx.xxx  |     1858176 |        1620872 |             2 |          0 |              0 |        0 |              0 |
|       2 |             65 | CONNECTED  | xxx.xxx.xxx.xxx  |   282893088 |       75385572 |             1 |          0 |              0 |        0 |              0 |
|       2 |             67 | CONNECTED  | xxx.xxx.xxx.xxx  |   110180948 |        8925352 |             2 |          0 |              0 |        0 |              0 |
|       2 |            231 | CONNECTING | xxx.xxx.xxx.xxx  |        9040 |            288 |             4 |          0 |              0 |        0 |              0 |
|       2 |              1 | CONNECTED  | -                | 63439655176 |    59699258336 |             0 |          0 |              0 |        0 |              0 |
|       2 |              3 | CONNECTING | -                |           0 |              0 |             0 |          0 |              0 |        0 |              0 |
|       2 |             66 | CONNECTING | -                |           0 |              0 |             0 |          0 |              0 |        0 |              0 |
|       2 |             68 | CONNECTED  | xxx.xxx.xxx.xxx  |     1876080 |        1619048 |             2 |          0 |              0 |        0 |              0 |
+---------+----------------+------------+----------------+-------------+----------------+---------------+------------+----------------+----------+----------------+

How many times have you seen 4009?
I can see 2 connect counts, so if you see it again you can check if the
connect count increases.

I am seeing the error quite a lot of times on running select statements…

the error shows up in the rest service and also via the MySQL CLI client and via an IDE as well

the connect count does not increase on seeing the error.

If you run another test, you can check to see which columns that increase
to see if you can see some pattern on what is going on.
From my horizon everything looks very good.

Could it be that your service interacts with another MySQL Server that isn’t
part of the cluster?
This MySQL Server would then reply with 4009.

Could happen e.g. if you have setup some load balancing service and one MySQL Server is not connected to the cluster.

nope. the service only talks to this one Mysql server only… no other database node is involved… (havent setup loadbalancing yet… wanted to get this one node working first)

when I run any query, the bytes received and bytes sent columns value increases… everything else remains unchanged.

Let me check if there is some way of getting 4009 without closing connection

Seems like we are at the end of what can be achieved without getting some more
debug info.

I will prepare a new mysqld binary which makes some amount of writing to the
MySQL error log when it hits things that makes it go into 4009 mode.

I will do that tomorrow and let you know when it is ready. Then you can run the same tests
with the new binary and hopefully the MySQL error log should contain some interesting
information that helps in understanding what goes on here.

sure that works… let me know when it is ready … i’ll set it up and run it on my end to capture more info so we can better debug.

thanks again for helping out by taking a look at this @mikaelronstrom

Now I have uploaded a new RonDB tarball to:
repo.hops.works/master/rondb-21.04.2-linux-glibc2.17-x86_64.tar.gz

The main change is in the mysqld which needs to be restarted with the new binary.
I have both updated the error messages and added new printouts in the MySQL error log.
You can find the new ones by grep’ing on RonDB.

Good luck, let me know the results.

@mikaelronstrom

we are getting the below error which is more descriptive:

Got error 4009 'No data node(s) available, check Cluster state' from NDBCLUSTER

on the MGM node on running show command i can see all nodes are connected properly.

the mysql_err.out file still does not have anything there no stacktrace no debug logs there…

all we have is the error response …