【算法检验】StepM和MSC
2021/9/18 22:11:08
本文主要是介绍【算法检验】StepM和MSC,对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
继上节SPA继续介绍StepM和MSC检验
【算法检验】SPA_Checkmate9949的博客-CSDN博客一、概念The multiple comparison procedures all allow for examining aspects of superior predictive ability.用于检验更优的预测能力There are three available:SPA - The test of Superior Predictive Ability, also known as the Reality Check (and accessible as RealityChechttps://blog.csdn.net/Checkmate9949/article/details/120370004
一、StepM检验
1、概念
SM类似于SPA,都具有同样的零假设。主要不同:SM可以识别比基准模型好的一系列模型,而不是只是验证某个模型更好。
Stepwise Multiple Testing is similar to the SPA and has the same null. The primary difference is that it identifies the set of models which are better than the benchmark, rather than just asking the basic question if any model is better.
2、例子
from arch.bootstrap import StepM stepm = StepM(bm_losses, model_losses) stepm.compute() #识别出优于基准模型的模型 print("Model indices:") print([model.split(".")[1] for model in stepm.superior_models]) #求各模型的平均Loss test2 = model_losses.mean(0) better_models = pd.concat([model_losses.mean(0), model_losses.mean(0)],axis=1) better_models.columns = ["Same or worse", "Better"] #得出better模型的index better = better_models.index.isin(stepm.superior_models) #得出worse模型的index worse = np.logical_not(better) #写为正确的值 better_models.loc[better, "Same or worse"] = np.nan better_models.loc[worse, "Better"] = np.nan #画图:绿色为更优的模型 fig = better_models.plot(style=["o", "s"], rot=270)
二、MSC检验
1、概念
The Model Confidence Set
MCS将一系列Loss作为输入,以发现类似的算法集。其主要输出为P值,其模型的P值高于Size。较小的P值代表模型能够被其他更好的模型拒绝。
The model confidence set takes a set of losses as its input and finds the set which are not statistically different from each other while controlling the familywise error rate. The primary output is a set of p-values, where models with a pvalue above the size are in the MCS. Small p-values indicate that the model is easily rejected from the set that includes the best.
2、例子
from arch.bootstrap import MCS # 取出25个算法,500/20=25 losses = model_losses.iloc[:, ::20] mcs = MCS(losses, size=0.10) mcs.compute() # 显示各模型的MSC P值 print("MCS P-values") print(mcs.pvalues) # Included: P>0.1 print("Included") included = mcs.included print([model.split(".")[1] for model in included]) # Excluded: P<=0.1 print("Excluded") excluded = mcs.excluded print([model.split(".")[1] for model in excluded])
Included为更优秀的算法集,P越大,算法越好。
这篇关于【算法检验】StepM和MSC的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!
- 2024-09-19Excel数据导出课程:初学者必备教程
- 2024-09-19Excel数据导入课程:新手入门指南
- 2024-09-19RBAC的权限管理入门教程
- 2024-09-19如何使用Svg Sprite Icon制作图标
- 2024-09-19uniapp 如何实现点赞后全局更新数据-icode9专业技术文章分享
- 2024-09-19云函数怎么运行wx-server-sdk-icode9专业技术文章分享
- 2024-09-19"dependencies": { "wx-server-sdk": "latest" },是什么意思-icode9专业技术文章分享
- 2024-09-16优化批处理流程:自定义BatchProcessorUtils的设计与应用
- 2024-09-15laravel collect游标批量插入的方法示例-icode9专业技术文章分享
- 2024-09-15怎么修改laravel的中间件 把获取IP改成X-Real-IP-icode9专业技术文章分享