This made a huge improvement to speculate decoding speeds on my B70 running SYCL. Below examples are for Muse Glimmer Q6_K (30B) running with DFlash2. NOTE: Due to the significantly reduced cost of token verification, the n-max was increased which is why the acceptance rate is lower. Before: ``` code_python pred= 192 draft= 127 acc= 106 rate=0.835 tok/s=38.0 code_cpp pred= 192 draft= 175 acc= 113 rate=0.646 tok/s=36.2 explain_concept pred= 192 draft= 135 acc= 116 rate=0.859 tok/s=41.2 summarize pred= 188 draft= 170 acc= 142 rate=0.835 tok/s=50.1 qa_factual pred= 192 draft= 145 acc= 107 rate=0.738 tok/s=37.0 translation pred= 192 draft= 175 acc= 133 rate=0.760 tok/s=45.1 creative_short pred= 192 draft= 130 acc= 105 rate=0.808 tok/s=37.1 stepwise_math pred= 192 draft= 167 acc= 145 rate=0.868 tok/s=53.2 long_code_review pred= 192 draft= 159 acc= 118 rate=0.742 tok/s=38.4 Aggregate: { "n_requests": 9, "total_predicted": 1724, "total_draft": 1383, "total_draft_accepted": 1085, "aggregate_accept_rate": 0.7845, "wall_s_total": 46.86 } ``` After: ``` code_python pred= 192 draft= 324 acc= 143 rate=0.441 tok/s=73.0 code_cpp pred= 192 draft= 424 acc= 140 rate=0.330 tok/s=68.5 explain_concept pred= 192 draft= 330 acc= 141 rate=0.427 tok/s=70.7 summarize pred= 192 draft= 416 acc= 151 rate=0.363 tok/s=85.4 qa_factual pred= 192 draft= 399 acc= 133 rate=0.333 tok/s=60.3 translation pred= 192 draft= 356 acc= 152 rate=0.427 tok/s=87.7 creative_short pred= 192 draft= 379 acc= 125 rate=0.330 tok/s=53.6 stepwise_math pred= 192 draft= 462 acc= 148 rate=0.320 tok/s=77.6 long_code_review pred= 192 draft= 414 acc= 139 rate=0.336 tok/s=64.2 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 3504, "total_draft_accepted": 1272, "aggregate_accept_rate": 0.363, "wall_s_total": 26.68 } ```   submitted by   /u/nickm_27 [link]   [comments]