Skip to content

Issues: open-compass/opencompass

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

180 Open 329 Closed
180 Open 329 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[Bug] SciCode miss with_background Bug Something isn't working
#1540 opened Sep 18, 2024 by tonysy
2 tasks done
@MaiziXiao
[Bug] mbpp_plus.jsonl 在哪里，下载的zip中没有这个东西
#1538 opened Sep 18, 2024 by kkwhale7
2 tasks done
@MaiziXiao
[Feature] About a subset of academic rankings
#1537 opened Sep 18, 2024 by mary-0830
1 task
@MaiziXiao
3
[Bug] PreTrainedTokenizerFast inference with unexpected parameter token_type_ids Bug Something isn't working
#1533 opened Sep 18, 2024 by kkwhale7
2 tasks done
@MaiziXiao
3
[Bug] MTBench数据集读取出现拼接bug
#1531 opened Sep 18, 2024 by zhang-junjian
2 tasks done
@MaiziXiao
2
使用Qwen2-72B-Base测试GPQA数据集时报错：NotImplementedError: OpenAI does not support ppl-based evaluation yet, try gen-based instead.
#1526 opened Sep 12, 2024 by 13416157913
2 tasks done
@acylam
2
[Bug] Cibench generate matplotlib has a wrong weights which should be [175, 2, 1, 156]
#1524 opened Sep 12, 2024 by imaginistLi
2 tasks done
@acylam
您好，请教一个问题，请问OpenCompass官网公开学术榜单中，测试的Qwen2模型中temperature、repetition_penalty、、top_p这些参数是设置多少的？模型部署的方式是采用什么方式部署的？
#1523 opened Sep 12, 2024 by 13416157913
2 tasks done
@acylam
3
[Feature] 数据集版本问题
#1509 opened Sep 9, 2024 by bjzhb666
1 task
@bittersweet1999
2
[Feature] How to config to support auto awq int4 llama3 model
#1506 opened Sep 9, 2024 by FlexLaughing
1 task
@bittersweet1999
Based on the openai service of the local model deployed by vllm, it is hoped that the sampling parameters top_p, etc. can be passed in when the model is initialized
#1505 opened Sep 9, 2024 by BrenchCC
1 task
@bittersweet1999
[Feature] 为什么调整batch_size之后，显存占用并没有提升呢
#1498 opened Sep 6, 2024 by belle9217
1 task
@tonysy
[Feature] 咱们oc啥时候能支持下lighteval的评测呀？
#1495 opened Sep 6, 2024 by caijie12138
1 task
@tonysy
2
[Bug] 单机多卡评测最后一张卡一直卡住
#1494 opened Sep 5, 2024 by bjzhb666
2 tasks done
4
[Feature] mtbench基线结果 backlog
#1492 opened Sep 5, 2024 by DietDietDiet
1 task
@acylam
1
[Feature] 添加mlc-llm后端支持 backlog
#1488 opened Sep 5, 2024 by XJY990705
1 task
@acylam @liushz
2
[Bug] 使用最新版本的glm4-9b-chat在Lawbench上推理会重复生成
#1477 opened Sep 2, 2024 by LKAMING97
2 tasks done
@bittersweet1999
17
[Bug] Error in Evaluation for HumanEval with pass@10
#1474 opened Sep 1, 2024 by Rcrossmeister
2 tasks done
@tonysy
1
[Bug] configs/datasets中path有的写的是相对路径，有的是Datasets_ids,会导致后续数据集加载错误，按目前版本都应该是Datasets_ids
#1471 opened Aug 29, 2024 by yikexiaoxing
2 tasks done
@acylam
1
[Feature] Support BigCodeBench
#1460 opened Aug 26, 2024 by terryyz
1 task
@bittersweet1999
2
我想知道如果我做了self consistent，是如何进行众多答案的确定和选择的
#1455 opened Aug 23, 2024 by BrenchCC
2 tasks done
@tonysy
[Feature] Support evaluation with vllm/lmdeploy openai-compatible api via CLI
#1446 opened Aug 22, 2024 by tonysy
1 task
@liushz
[Feature] Provide specific prompt version for common datasets in README
#1445 opened Aug 22, 2024 by tonysy
1 task done
@acylam
[Feature] Update the doc for introduction the evaluation with num_worker and multi-gpu
#1444 opened Aug 22, 2024 by tonysy
1 task
@tonysy
[Feature] Support SGLang and Ollama
#1443 opened Aug 22, 2024 by tonysy
1 task
@liushz
ProTip! Follow long discussions with comments:>50.