2023-07-20 16:53:15,180 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-07-20 16:53:15,500 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-07-20 16:53:15,612 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-07-20 16:53:15,679 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-20 16:53:15,687 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-20 16:53:15,705 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=310, ProcessCount=173, AvailableMemoryMB=7234
2023-07-20 16:53:15,707 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-20 16:53:15,708 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-20 16:53:15,867 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004378559:84 , srv1025608422:84 , srv1065883994:84 , srv1090693678:84 , srv1099273728:84 , srv115931915:84 , srv1163847581:84 , srv1177192053:84 , srv1179980377:84 , srv1185313560:84 , srv1191848301:84 , srv1193419245:84 , srv1240997118:84 , srv1248245334:84 , srv1259817505:84 , srv1307981771:84 , srv1320671923:84 , srv1329624616:84 , srv1344618135:84 , srv1345841498:84 , srv1355569437:84 , srv1363830561:84 , srv1377433608:84 , srv1382928041:84 , srv1390192671:84 , srv1413954963:84 , srv1425586212:84 , srv1455985742:84 , srv1464723508:84 , srv1472993960:84 , srv1480923531:84 , srv148586181:84 , srv148909924:84 , srv1530477564:84 , srv1563231718:84 , srv1571761942:84 , srv1583924260:84 , srv1585387806:84 , srv1587815853:84 , srv1592973281:84 , srv1619129959:84 , srv1633581654:84 , srv165151077:84 , srv16581645:84 , srv1660097051:84 , srv1691330649:84 , srv1710698304:84 , srv1757625461:84 , srv17822742:84 , srv1840349653:84 , srv1895286245:84 , srv1916978369:84 , srv1920961563:84 , srv1947170465:84 , srv1963442779:84 , srv198386147:84 , srv1990958659:84 , srv2003616583:84 , srv200427369:84 , srv2010637181:84 , srv2020036251:84 , srv2024154551:84 , srv2035850771:84 , srv2036398347:84 , srv2060297799:84 , srv2083367839:84 , srv2102743320:84 , srv2141986506:84 , srv2146068750:84 , srv237123824:84 , srv242178276:84 , srv267162296:84 , srv28221441:84 , srv325471328:84 , srv34610033:84 , srv348163074:84 , srv35454939:84 , srv383911642:84 , srv421144188:84 , srv454422080:84 , srv472818993:84 , srv477232900:84 , srv505856095:84 , srv520313835:84 , srv521703837:84 , srv538972711:84 , srv554617228:84 , srv563716777:84 , srv58221602:84 , srv59369126:84 , srv626472471:84 , srv658504788:84 , srv738134095:84 , srv743985832:84 , srv753710912:84 , srv871793799:84 , srv876320759:84 , srv905310161:84 , srv947628267:84 , srv832751469:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-07-20 16:53:15,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-20 16:53:15,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090693678=3, srv743985832=93, srv2010637181=59, srv1355569437=20, srv1329624616=17, srv1307981771=15, srv198386147=55, srv472818993=80, srv505856095=82, srv1947170465=53, srv383911642=77, srv871793799=96, srv2102743320=66, srv115931915=5, srv148909924=32, srv237123824=69, srv59369126=89, srv1455985742=27, srv905310161=98, srv1990958659=56, srv477232900=81, srv1240997118=12, srv1571761942=35, srv538972711=85, srv2146068750=68, srv1895286245=50, srv1390192671=24, srv2035850771=62, srv753710912=94, srv1633581654=41, srv626472471=90, srv1193419245=11, srv1619129959=40, srv1916978369=51, srv520313835=83, srv832751469=95, srv1345841498=19, srv1530477564=33, srv1377433608=22, srv1710698304=46, srv1592973281=39, srv1004378559=0, srv658504788=91, srv1840349653=49, srv1963442779=54, srv1025608422=1, srv1177192053=7, srv2036398347=63, srv1382928041=23, srv2020036251=60, srv1413954963=25, srv738134095=92, srv1464723508=28, srv1585387806=37, srv1472993960=29, srv165151077=42, srv267162296=71, srv28221441=72, srv58221602=88, srv16581645=43, srv1344618135=18, srv325471328=73, srv947628267=99, srv1248245334=13, srv521703837=84, srv1363830561=21, srv34610033=74, srv1587815853=38, srv1920961563=52, srv1320671923=16, srv2060297799=64, srv1583924260=36, srv2141986506=67, srv35454939=76, srv1425586212=26, srv242178276=70, srv1185313560=9, srv1163847581=6, srv1480923531=30, srv876320759=97, srv2083367839=65, srv1259817505=14, srv1563231718=34, srv2024154551=61, srv348163074=75, srv1691330649=45, srv1191848301=10, srv148586181=31, srv1660097051=44, srv1179980377=8, srv554617228=86, srv17822742=48, srv454422080=79, srv1757625461=47, srv200427369=58, srv421144188=78, srv1099273728=4, srv2003616583=57, srv563716777=87, srv1065883994=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-20 16:53:15,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-20 16:53:15,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-20 16:53:15,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-20 16:53:15,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-20 16:53:15,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-20 16:53:15,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-20 16:53:15,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-20 16:53:15,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-20 16:53:15,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-20 16:53:15,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-20 16:53:15,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-20 16:53:15,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-20 16:53:15,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-20 16:53:15,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-20 16:53:15,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-20 16:53:15,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-20 16:53:15,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-20 16:53:15,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-20 16:53:15,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-20 16:53:15,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-20 16:53:15,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-20 16:53:15,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-20 16:53:15,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-20 16:53:15,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-20 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-20 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-20 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-20 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-20 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-20 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-20 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-20 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-20 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-20 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-20 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-20 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-20 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-20 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-20 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-20 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-20 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-20 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-20 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-20 16:53:15,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-20 16:53:15,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-20 16:53:15,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-20 16:53:15,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-20 16:53:15,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-20 16:53:15,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-20 16:53:15,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-20 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-20 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-20 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-20 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-20 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-20 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-20 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-20 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-20 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-20 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-20 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-20 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-20 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-20 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-20 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-20 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-20 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-20 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-20 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-20 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-20 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-20 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-20 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-20 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-20 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-20 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-20 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-20 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-20 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-20 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-20 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-20 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-20 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-20 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-20 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-20 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-20 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-20 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-20 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-20 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-20 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-20 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-20 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-20 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-20 16:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-20 16:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-20 16:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-20 16:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-20 16:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-20 16:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-20 16:53:16,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-20 16:53:16,217 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:16,217 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-20 16:53:16,218 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-20 16:53:16,639 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 608 ms to try 5000 different iterations.  Found a solution that moves 4672 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17881490609439665. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039470873679980534, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05327859143478712, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5191111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039470873679980534, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23941666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:16,664 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1191848301:82 , srv1320671923:82 , srv17822742:82 , srv1363830561:83 , srv1563231718:83 , srv1587815853:83 , srv2141986506:83 , srv421144188:83 , srv1065883994:84 , srv115931915:84 , srv1163847581:84 , srv1177192053:84 , srv1920961563:84 , srv1963442779:84 , srv237123824:84 , srv563716777:84 , srv905310161:84 , srv1025608422:85 , srv1099273728:85 , srv1248245334:85 , srv1259817505:85 , srv1345841498:85 , srv1355569437:85 , srv1413954963:85 , srv1425586212:85 , srv1633581654:85 , srv1840349653:85 , srv1947170465:85 , srv1990958659:85 , srv2003616583:85 , srv200427369:85 , srv2035850771:85 , srv325471328:85 , srv34610033:85 , srv505856095:85 , srv626472471:85 , srv1179980377:86 , srv1193419245:86 , srv1240997118:86 , srv1480923531:86 , srv1530477564:86 , srv1571761942:86 , srv1583924260:86 , srv1585387806:86 , srv1619129959:86 , srv198386147:86 , srv242178276:86 , srv267162296:86 , srv28221441:86 , srv472818993:86 , srv477232900:86 , srv58221602:86 , srv753710912:86 , srv876320759:86 , srv1090693678:87 , srv1307981771:87 , srv1329624616:87 , srv1377433608:87 , srv1592973281:87 , srv1660097051:87 , srv1757625461:87 , srv1895286245:87 , srv2010637181:87 , srv2036398347:87 , srv2060297799:87 , srv2083367839:87 , srv454422080:87 , srv521703837:87 , srv554617228:87 , srv1004378559:88 , srv1185313560:88 , srv1344618135:88 , srv1455985742:88 , srv1464723508:88 , srv1472993960:88 , srv165151077:88 , srv1710698304:88 , srv2020036251:88 , srv2024154551:88 , srv658504788:88 , srv743985832:88 , srv16581645:89 , srv1691330649:89 , srv1916978369:89 , srv2146068750:89 , srv538972711:89 , srv59369126:89 , srv738134095:89 , srv871793799:89 , srv947628267:89 , srv1382928041:90 , srv1390192671:90 , srv348163074:90 , srv35454939:90 , srv520313835:90 , srv148586181:91 , srv383911642:91 , srv148909924:92 , srv2102743320:93 , srv832751469:441 }
2023-07-20 16:53:16,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-20 16:53:16,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090693678=3, srv743985832=93, srv2010637181=59, srv1355569437=20, srv1329624616=17, srv1307981771=15, srv198386147=55, srv472818993=80, srv505856095=82, srv1947170465=53, srv383911642=77, srv871793799=96, srv2102743320=66, srv115931915=5, srv148909924=32, srv237123824=69, srv59369126=89, srv1455985742=27, srv905310161=98, srv1990958659=56, srv477232900=81, srv1240997118=12, srv1571761942=35, srv538972711=85, srv2146068750=68, srv1895286245=50, srv1390192671=24, srv2035850771=62, srv753710912=94, srv1633581654=41, srv626472471=90, srv1193419245=11, srv1619129959=40, srv1916978369=51, srv520313835=83, srv832751469=95, srv1345841498=19, srv1530477564=33, srv1377433608=22, srv1710698304=46, srv1592973281=39, srv1004378559=0, srv658504788=91, srv1840349653=49, srv1963442779=54, srv1025608422=1, srv1177192053=7, srv2036398347=63, srv1382928041=23, srv2020036251=60, srv1413954963=25, srv738134095=92, srv1464723508=28, srv1585387806=37, srv1472993960=29, srv165151077=42, srv267162296=71, srv28221441=72, srv58221602=88, srv16581645=43, srv1344618135=18, srv325471328=73, srv947628267=99, srv1248245334=13, srv521703837=84, srv1363830561=21, srv34610033=74, srv1587815853=38, srv1920961563=52, srv1320671923=16, srv2060297799=64, srv1583924260=36, srv2141986506=67, srv35454939=76, srv1425586212=26, srv242178276=70, srv1185313560=9, srv1163847581=6, srv1480923531=30, srv876320759=97, srv2083367839=65, srv1259817505=14, srv1563231718=34, srv2024154551=61, srv348163074=75, srv1691330649=45, srv1191848301=10, srv148586181=31, srv1660097051=44, srv1179980377=8, srv554617228=86, srv17822742=48, srv454422080=79, srv1757625461=47, srv200427369=58, srv421144188=78, srv1099273728=4, srv2003616583=57, srv563716777=87, srv1065883994=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-20 16:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-20 16:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-20 16:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-20 16:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-20 16:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-20 16:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-20 16:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-20 16:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-20 16:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-20 16:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-20 16:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-20 16:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-20 16:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-20 16:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-20 16:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-20 16:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-20 16:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-20 16:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-20 16:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-20 16:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-20 16:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-20 16:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-20 16:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-20 16:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-20 16:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-20 16:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-20 16:53:16,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-20 16:53:16,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-20 16:53:16,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-20 16:53:16,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-20 16:53:16,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-20 16:53:16,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-20 16:53:16,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-20 16:53:16,842 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039470873679980534, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05327859143478712, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039470873679980534, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23941666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:16,842 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-20 16:53:16,843 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1787821920511854, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039470873679980534, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05327859143478712, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039470873679980534, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23941666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-20 16:53:17,201 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 464 ms to try 5000 different iterations.  Found a solution that moves 2633 regions; Going from a computed imbalance of 0.1787821920511854 to a new imbalance of 0.01646142313726719. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020587456666415115); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0308466392256134, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29255555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020587456666415115); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:17,215 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2102743320:79 , srv520313835:80 , srv348163074:81 , srv35454939:81 , srv1390192671:82 , srv1464723508:82 , srv1472993960:82 , srv148586181:82 , srv148909924:82 , srv1916978369:82 , srv59369126:82 , srv738134095:82 , srv871793799:82 , srv1592973281:83 , srv16581645:83 , srv1691330649:83 , srv383911642:83 , srv472818993:83 , srv538972711:83 , srv658504788:83 , srv1185313560:84 , srv1307981771:84 , srv1377433608:84 , srv1382928041:84 , srv1455985742:84 , srv1571761942:84 , srv165151077:84 , srv1895286245:84 , srv2010637181:84 , srv28221441:84 , srv521703837:84 , srv554617228:84 , srv1004378559:85 , srv1193419245:85 , srv1259817505:85 , srv1344618135:85 , srv1710698304:85 , srv2020036251:85 , srv2060297799:85 , srv2146068750:85 , srv454422080:85 , srv477232900:85 , srv505856095:85 , srv947628267:85 , srv1025608422:86 , srv1090693678:86 , srv1240997118:86 , srv1355569437:86 , srv1480923531:86 , srv1585387806:86 , srv2024154551:86 , srv2036398347:86 , srv58221602:86 , srv743985832:86 , srv753710912:86 , srv876320759:86 , srv1248245334:87 , srv1530477564:87 , srv1583924260:87 , srv1619129959:87 , srv1633581654:87 , srv1660097051:87 , srv1757625461:87 , srv1947170465:87 , srv1963442779:87 , srv198386147:87 , srv2003616583:87 , srv200427369:87 , srv2035850771:87 , srv2083367839:87 , srv2141986506:87 , srv242178276:87 , srv267162296:87 , srv325471328:87 , srv626472471:87 , srv115931915:88 , srv1177192053:88 , srv1179980377:88 , srv1329624616:88 , srv1345841498:88 , srv1413954963:88 , srv1425586212:88 , srv1563231718:88 , srv1587815853:88 , srv1920961563:88 , srv1990958659:88 , srv237123824:88 , srv563716777:88 , srv905310161:88 , srv1065883994:89 , srv1099273728:89 , srv1163847581:89 , srv1363830561:89 , srv1840349653:89 , srv34610033:89 , srv17822742:90 , srv1320671923:91 , srv1191848301:92 , srv421144188:92 , srv832751469:516 }
2023-07-20 16:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-20 16:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090693678=3, srv743985832=93, srv2010637181=59, srv1355569437=20, srv1329624616=17, srv1307981771=15, srv198386147=55, srv472818993=80, srv505856095=82, srv1947170465=53, srv383911642=77, srv871793799=96, srv2102743320=66, srv115931915=5, srv148909924=32, srv237123824=69, srv59369126=89, srv1455985742=27, srv905310161=98, srv1990958659=56, srv477232900=81, srv1240997118=12, srv1571761942=35, srv538972711=85, srv2146068750=68, srv1895286245=50, srv1390192671=24, srv2035850771=62, srv753710912=94, srv1633581654=41, srv626472471=90, srv1193419245=11, srv1619129959=40, srv1916978369=51, srv520313835=83, srv832751469=95, srv1345841498=19, srv1530477564=33, srv1377433608=22, srv1710698304=46, srv1592973281=39, srv1004378559=0, srv658504788=91, srv1840349653=49, srv1963442779=54, srv1025608422=1, srv1177192053=7, srv2036398347=63, srv1382928041=23, srv2020036251=60, srv1413954963=25, srv738134095=92, srv1464723508=28, srv1585387806=37, srv1472993960=29, srv165151077=42, srv267162296=71, srv28221441=72, srv58221602=88, srv16581645=43, srv1344618135=18, srv325471328=73, srv947628267=99, srv1248245334=13, srv521703837=84, srv1363830561=21, srv34610033=74, srv1587815853=38, srv1920961563=52, srv1320671923=16, srv2060297799=64, srv1583924260=36, srv2141986506=67, srv35454939=76, srv1425586212=26, srv242178276=70, srv1185313560=9, srv1163847581=6, srv1480923531=30, srv876320759=97, srv2083367839=65, srv1259817505=14, srv1563231718=34, srv2024154551=61, srv348163074=75, srv1691330649=45, srv1191848301=10, srv148586181=31, srv1660097051=44, srv1179980377=8, srv554617228=86, srv17822742=48, srv454422080=79, srv1757625461=47, srv200427369=58, srv421144188=78, srv1099273728=4, srv2003616583=57, srv563716777=87, srv1065883994=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-20 16:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-20 16:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-20 16:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-20 16:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-20 16:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-20 16:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-20 16:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-20 16:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-20 16:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-20 16:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-20 16:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-20 16:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-20 16:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-20 16:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-20 16:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-20 16:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-20 16:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-20 16:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-20 16:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-20 16:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-20 16:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-20 16:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-20 16:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-20 16:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-20 16:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-20 16:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-20 16:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-20 16:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-20 16:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-20 16:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-20 16:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-20 16:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-20 16:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-20 16:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-20 16:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-20 16:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-20 16:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-20 16:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-20 16:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-20 16:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-20 16:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-20 16:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-20 16:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-20 16:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-20 16:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-20 16:53:17,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-20 16:53:17,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-20 16:53:17,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-20 16:53:17,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-20 16:53:17,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-20 16:53:17,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-20 16:53:17,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-20 16:53:17,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-20 16:53:17,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-20 16:53:17,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-20 16:53:17,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-20 16:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-20 16:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-20 16:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-20 16:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-20 16:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-20 16:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-20 16:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-20 16:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-20 16:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-20 16:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-20 16:53:17,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-20 16:53:17,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-20 16:53:17,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-20 16:53:17,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-20 16:53:17,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-20 16:53:17,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-20 16:53:17,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-20 16:53:17,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-20 16:53:17,413 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020587456666415115); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0308466392256134, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020587456666415115); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:17,413 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-20 16:53:17,413 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016442986477212554, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020587456666415115); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0308466392256134, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020587456666415115); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-20 16:53:17,775 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 505 ms to try 5000 different iterations.  Found a solution that moves 994 regions; Going from a computed imbalance of 0.016442986477212554 to a new imbalance of 8.216150940209206E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004761423737128353); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007550800642840709); MoveCostFunction : (multiplier=7.0, imbalance=0.11044444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004761423737128353); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:17,777 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1099273728:83 , srv1179980377:83 , srv1191848301:83 , srv1329624616:83 , srv1382928041:83 , srv2024154551:83 , srv2083367839:83 , srv34610033:83 , srv421144188:83 , srv1320671923:84 , srv1413954963:84 , srv148909924:84 , srv1660097051:84 , srv1840349653:84 , srv2036398347:84 , srv2146068750:84 , srv267162296:84 , srv383911642:84 , srv743985832:84 , srv947628267:84 , srv1004378559:85 , srv1065883994:85 , srv1090693678:85 , srv1185313560:85 , srv1248245334:85 , srv1344618135:85 , srv1345841498:85 , srv1455985742:85 , srv148586181:85 , srv1530477564:85 , srv1583924260:85 , srv1619129959:85 , srv1633581654:85 , srv165151077:85 , srv16581645:85 , srv1691330649:85 , srv1710698304:85 , srv1757625461:85 , srv17822742:85 , srv1947170465:85 , srv198386147:85 , srv1990958659:85 , srv2003616583:85 , srv2020036251:85 , srv2035850771:85 , srv237123824:85 , srv325471328:85 , srv454422080:85 , srv563716777:85 , srv905310161:85 , srv1025608422:86 , srv115931915:86 , srv1163847581:86 , srv1177192053:86 , srv1193419245:86 , srv1240997118:86 , srv1307981771:86 , srv1363830561:86 , srv1377433608:86 , srv1390192671:86 , srv1425586212:86 , srv1480923531:86 , srv1563231718:86 , srv1585387806:86 , srv1587815853:86 , srv1916978369:86 , srv1920961563:86 , srv1963442779:86 , srv200427369:86 , srv2060297799:86 , srv2102743320:86 , srv242178276:86 , srv348163074:86 , srv35454939:86 , srv477232900:86 , srv538972711:86 , srv554617228:86 , srv58221602:86 , srv658504788:86 , srv753710912:86 , srv871793799:86 , srv876320759:86 , srv1355569437:87 , srv1464723508:87 , srv1472993960:87 , srv1571761942:87 , srv1592973281:87 , srv1895286245:87 , srv2010637181:87 , srv2141986506:87 , srv28221441:87 , srv505856095:87 , srv521703837:87 , srv626472471:87 , srv738134095:87 , srv1259817505:88 , srv520313835:88 , srv59369126:88 , srv472818993:89 , srv832751469:543 }
2023-07-20 16:53:17,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-20 16:53:17,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090693678=3, srv743985832=93, srv2010637181=59, srv1355569437=20, srv1329624616=17, srv1307981771=15, srv198386147=55, srv472818993=80, srv505856095=82, srv1947170465=53, srv383911642=77, srv871793799=96, srv2102743320=66, srv115931915=5, srv148909924=32, srv237123824=69, srv59369126=89, srv1455985742=27, srv905310161=98, srv1990958659=56, srv477232900=81, srv1240997118=12, srv1571761942=35, srv538972711=85, srv2146068750=68, srv1895286245=50, srv1390192671=24, srv2035850771=62, srv753710912=94, srv1633581654=41, srv626472471=90, srv1193419245=11, srv1619129959=40, srv1916978369=51, srv520313835=83, srv832751469=95, srv1345841498=19, srv1530477564=33, srv1377433608=22, srv1710698304=46, srv1592973281=39, srv1004378559=0, srv658504788=91, srv1840349653=49, srv1963442779=54, srv1025608422=1, srv1177192053=7, srv2036398347=63, srv1382928041=23, srv2020036251=60, srv1413954963=25, srv738134095=92, srv1464723508=28, srv1585387806=37, srv1472993960=29, srv165151077=42, srv267162296=71, srv28221441=72, srv58221602=88, srv16581645=43, srv1344618135=18, srv325471328=73, srv947628267=99, srv1248245334=13, srv521703837=84, srv1363830561=21, srv34610033=74, srv1587815853=38, srv1920961563=52, srv1320671923=16, srv2060297799=64, srv1583924260=36, srv2141986506=67, srv35454939=76, srv1425586212=26, srv242178276=70, srv1185313560=9, srv1163847581=6, srv1480923531=30, srv876320759=97, srv2083367839=65, srv1259817505=14, srv1563231718=34, srv2024154551=61, srv348163074=75, srv1691330649=45, srv1191848301=10, srv148586181=31, srv1660097051=44, srv1179980377=8, srv554617228=86, srv17822742=48, srv454422080=79, srv1757625461=47, srv200427369=58, srv421144188=78, srv1099273728=4, srv2003616583=57, srv563716777=87, srv1065883994=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-20 16:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-20 16:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-20 16:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-20 16:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-20 16:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-20 16:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-20 16:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-20 16:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-20 16:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-20 16:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-20 16:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-20 16:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-20 16:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-20 16:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-20 16:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-20 16:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-20 16:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-20 16:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-20 16:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-20 16:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-20 16:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-20 16:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-20 16:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-20 16:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-20 16:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-20 16:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-20 16:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-20 16:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-20 16:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-20 16:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-20 16:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-20 16:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-20 16:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-20 16:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-20 16:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-20 16:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-20 16:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-20 16:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-20 16:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-20 16:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-20 16:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-20 16:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-20 16:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-20 16:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-20 16:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-20 16:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-20 16:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-20 16:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-20 16:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-20 16:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-20 16:53:17,940 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004761423737128353); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007550800642840709); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004761423737128353); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:17,940 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-20 16:53:17,940 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.146549572589348E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004761423737128353); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007550800642840709); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004761423737128353); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-20 16:53:18,302 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 470 ms to try 5000 different iterations.  Found a solution that moves 228 regions; Going from a computed imbalance of 8.146549572589348E-4 to a new imbalance of 1.9916149807506773E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.025333333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:18,303 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1163847581:83 , srv1425586212:83 , srv2010637181:83 , srv242178276:83 , srv59369126:83 , srv626472471:83 , srv1004378559:84 , srv1065883994:84 , srv1090693678:84 , srv115931915:84 , srv1177192053:84 , srv1240997118:84 , srv1259817505:84 , srv1329624616:84 , srv1344618135:84 , srv1345841498:84 , srv1355569437:84 , srv1382928041:84 , srv1390192671:84 , srv1480923531:84 , srv148586181:84 , srv148909924:84 , srv1530477564:84 , srv1585387806:84 , srv1619129959:84 , srv16581645:84 , srv1660097051:84 , srv1710698304:84 , srv1757625461:84 , srv1840349653:84 , srv1895286245:84 , srv1920961563:84 , srv198386147:84 , srv1990958659:84 , srv200427369:84 , srv2020036251:84 , srv2060297799:84 , srv2102743320:84 , srv2146068750:84 , srv383911642:84 , srv421144188:84 , srv472818993:84 , srv520313835:84 , srv521703837:84 , srv538972711:84 , srv58221602:84 , srv658504788:84 , srv738134095:84 , srv743985832:84 , srv753710912:84 , srv876320759:84 , srv947628267:84 , srv1025608422:85 , srv1099273728:85 , srv1179980377:85 , srv1185313560:85 , srv1191848301:85 , srv1193419245:85 , srv1248245334:85 , srv1307981771:85 , srv1320671923:85 , srv1363830561:85 , srv1413954963:85 , srv1455985742:85 , srv1464723508:85 , srv1472993960:85 , srv1563231718:85 , srv1571761942:85 , srv1583924260:85 , srv1587815853:85 , srv1592973281:85 , srv1633581654:85 , srv165151077:85 , srv1691330649:85 , srv17822742:85 , srv1916978369:85 , srv1947170465:85 , srv1963442779:85 , srv2003616583:85 , srv2024154551:85 , srv2035850771:85 , srv2036398347:85 , srv2083367839:85 , srv2141986506:85 , srv237123824:85 , srv267162296:85 , srv28221441:85 , srv325471328:85 , srv34610033:85 , srv348163074:85 , srv35454939:85 , srv454422080:85 , srv477232900:85 , srv505856095:85 , srv554617228:85 , srv563716777:85 , srv871793799:85 , srv905310161:85 , srv1377433608:86 , srv832751469:642 }
2023-07-20 16:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-20 16:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090693678=3, srv743985832=93, srv2010637181=59, srv1355569437=20, srv1329624616=17, srv1307981771=15, srv198386147=55, srv472818993=80, srv505856095=82, srv1947170465=53, srv383911642=77, srv871793799=96, srv2102743320=66, srv115931915=5, srv148909924=32, srv237123824=69, srv59369126=89, srv1455985742=27, srv905310161=98, srv1990958659=56, srv477232900=81, srv1240997118=12, srv1571761942=35, srv538972711=85, srv2146068750=68, srv1895286245=50, srv1390192671=24, srv2035850771=62, srv753710912=94, srv1633581654=41, srv626472471=90, srv1193419245=11, srv1619129959=40, srv1916978369=51, srv520313835=83, srv832751469=95, srv1345841498=19, srv1530477564=33, srv1377433608=22, srv1710698304=46, srv1592973281=39, srv1004378559=0, srv658504788=91, srv1840349653=49, srv1963442779=54, srv1025608422=1, srv1177192053=7, srv2036398347=63, srv1382928041=23, srv2020036251=60, srv1413954963=25, srv738134095=92, srv1464723508=28, srv1585387806=37, srv1472993960=29, srv165151077=42, srv267162296=71, srv28221441=72, srv58221602=88, srv16581645=43, srv1344618135=18, srv325471328=73, srv947628267=99, srv1248245334=13, srv521703837=84, srv1363830561=21, srv34610033=74, srv1587815853=38, srv1920961563=52, srv1320671923=16, srv2060297799=64, srv1583924260=36, srv2141986506=67, srv35454939=76, srv1425586212=26, srv242178276=70, srv1185313560=9, srv1163847581=6, srv1480923531=30, srv876320759=97, srv2083367839=65, srv1259817505=14, srv1563231718=34, srv2024154551=61, srv348163074=75, srv1691330649=45, srv1191848301=10, srv148586181=31, srv1660097051=44, srv1179980377=8, srv554617228=86, srv17822742=48, srv454422080=79, srv1757625461=47, srv200427369=58, srv421144188=78, srv1099273728=4, srv2003616583=57, srv563716777=87, srv1065883994=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-20 16:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-20 16:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-20 16:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-20 16:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-20 16:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-20 16:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-20 16:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-20 16:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-20 16:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-20 16:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-20 16:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-20 16:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-20 16:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-20 16:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-20 16:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-20 16:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-20 16:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-20 16:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-20 16:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-20 16:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-20 16:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-20 16:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-20 16:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-20 16:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-20 16:53:18,455 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8319659685039178E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:18,455 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1163847581:83 , srv1425586212:83 , srv2010637181:83 , srv242178276:83 , srv59369126:83 , srv626472471:83 , srv1004378559:84 , srv1065883994:84 , srv1090693678:84 , srv115931915:84 , srv1177192053:84 , srv1240997118:84 , srv1259817505:84 , srv1329624616:84 , srv1344618135:84 , srv1345841498:84 , srv1355569437:84 , srv1382928041:84 , srv1390192671:84 , srv1480923531:84 , srv148586181:84 , srv148909924:84 , srv1530477564:84 , srv1585387806:84 , srv1619129959:84 , srv16581645:84 , srv1660097051:84 , srv1710698304:84 , srv1757625461:84 , srv1840349653:84 , srv1895286245:84 , srv1920961563:84 , srv198386147:84 , srv1990958659:84 , srv200427369:84 , srv2020036251:84 , srv2060297799:84 , srv2102743320:84 , srv2146068750:84 , srv383911642:84 , srv421144188:84 , srv472818993:84 , srv520313835:84 , srv521703837:84 , srv538972711:84 , srv58221602:84 , srv658504788:84 , srv738134095:84 , srv743985832:84 , srv753710912:84 , srv876320759:84 , srv947628267:84 , srv1025608422:85 , srv1099273728:85 , srv1179980377:85 , srv1185313560:85 , srv1191848301:85 , srv1193419245:85 , srv1248245334:85 , srv1307981771:85 , srv1320671923:85 , srv1363830561:85 , srv1413954963:85 , srv1455985742:85 , srv1464723508:85 , srv1472993960:85 , srv1563231718:85 , srv1571761942:85 , srv1583924260:85 , srv1587815853:85 , srv1592973281:85 , srv1633581654:85 , srv165151077:85 , srv1691330649:85 , srv17822742:85 , srv1916978369:85 , srv1947170465:85 , srv1963442779:85 , srv2003616583:85 , srv2024154551:85 , srv2035850771:85 , srv2036398347:85 , srv2083367839:85 , srv2141986506:85 , srv237123824:85 , srv267162296:85 , srv28221441:85 , srv325471328:85 , srv34610033:85 , srv348163074:85 , srv35454939:85 , srv454422080:85 , srv477232900:85 , srv505856095:85 , srv554617228:85 , srv563716777:85 , srv871793799:85 , srv905310161:85 , srv1377433608:86 , srv832751469:642 }
2023-07-20 16:53:18,495 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=333 (was 310) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=6435 (was 7234)
2023-07-20 16:53:18,506 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=333, ProcessCount=173, AvailableMemoryMB=6433
2023-07-20 16:53:18,507 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-20 16:53:18,507 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-07-20 16:53:18,520 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1032069739:84 , srv1084653439:84 , srv1252116269:84 , srv1298521524:84 , srv138464771:84 , srv1409962376:84 , srv1427278259:84 , srv1568503410:84 , srv1583344317:84 , srv1602838796:84 , srv1775822941:84 , srv1830919541:84 , srv1846125923:84 , srv2025405988:84 , srv205831763:84 , srv2091366087:84 , srv2104852477:84 , srv233384656:84 , srv299606468:84 , srv315885984:84 , srv400732078:84 , srv458245684:84 , srv518920905:84 , srv655195067:84 , srv708304273:84 , srv738477230:84 , srv803969712:84 , srv924704173:84 , srv931329258:84 , srv548999341:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-07-20 16:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-20 16:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv138464771=4, srv315885984=19, srv1409962376=5, srv924704173=28, srv233384656=17, srv738477230=26, srv205831763=14, srv1846125923=12, srv518920905=22, srv931329258=29, srv400732078=20, srv299606468=18, srv655195067=24, srv708304273=25, srv1252116269=2, srv1568503410=7, srv548999341=23, srv803969712=27, srv1032069739=0, srv1084653439=1, srv1583344317=8, srv1298521524=3, srv1830919541=11, srv1775822941=10, srv2025405988=13, srv2091366087=15, srv458245684=21, srv2104852477=16, srv1602838796=9, srv1427278259=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-20 16:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-20 16:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-20 16:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-20 16:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-20 16:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-20 16:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-20 16:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-20 16:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-20 16:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-20 16:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-20 16:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-20 16:53:18,569 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.103283182271879, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:53:18,569 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912518485199112, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.103283182271879, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-07-20 16:55:18,544 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5494490 different iterations.  Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912518485199112 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:55:18,550 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1032069739:90 , srv1084653439:90 , srv1252116269:90 , srv1298521524:90 , srv138464771:90 , srv1409962376:90 , srv1427278259:90 , srv1568503410:90 , srv1583344317:90 , srv1602838796:90 , srv1775822941:90 , srv1830919541:90 , srv1846125923:90 , srv2025405988:90 , srv205831763:90 , srv2091366087:90 , srv2104852477:90 , srv233384656:90 , srv299606468:90 , srv315885984:90 , srv400732078:90 , srv458245684:90 , srv518920905:90 , srv548999341:90 , srv655195067:90 , srv708304273:90 , srv738477230:90 , srv803969712:90 , srv924704173:90 , srv931329258:90 }
2023-07-20 16:55:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-20 16:55:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv138464771=4, srv315885984=19, srv1409962376=5, srv924704173=28, srv233384656=17, srv738477230=26, srv205831763=14, srv1846125923=12, srv518920905=22, srv931329258=29, srv400732078=20, srv299606468=18, srv655195067=24, srv708304273=25, srv1252116269=2, srv1568503410=7, srv548999341=23, srv803969712=27, srv1032069739=0, srv1084653439=1, srv1583344317=8, srv1298521524=3, srv1830919541=11, srv1775822941=10, srv2025405988=13, srv2091366087=15, srv458245684=21, srv2104852477=16, srv1602838796=9, srv1427278259=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-20 16:55:18,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-20 16:55:18,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-20 16:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-20 16:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-20 16:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-20 16:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-20 16:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-20 16:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-20 16:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-20 16:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-20 16:55:18,568 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-20 16:55:18,568 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1032069739:90 , srv1084653439:90 , srv1252116269:90 , srv1298521524:90 , srv138464771:90 , srv1409962376:90 , srv1427278259:90 , srv1568503410:90 , srv1583344317:90 , srv1602838796:90 , srv1775822941:90 , srv1830919541:90 , srv1846125923:90 , srv2025405988:90 , srv205831763:90 , srv2091366087:90 , srv2104852477:90 , srv233384656:90 , srv299606468:90 , srv315885984:90 , srv400732078:90 , srv458245684:90 , srv518920905:90 , srv548999341:90 , srv655195067:90 , srv708304273:90 , srv738477230:90 , srv803969712:90 , srv924704173:90 , srv931329258:90 }
2023-07-20 16:55:18,581 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=187 (was 333), ProcessCount=173 (was 173), AvailableMemoryMB=5515 (was 6433)