** We apologize if you receive multiple copies of this CfP **
** For the online version of this Call, visit:
https://recsys.acm.org/recsys23/call/reproducibility **
The reproducibility of empirical results is a cornerstone of scientific research and a
prerequisite to ensure continuous progress in our field. ACM RecSys ‘23 therefore strongly
encourages submissions which repeat or analyze prior works in similar or alternative
settings or which contribute to an increased level of reproducibility in the future.
Types of papers
We encourage the submission of different types of papers, including:
* Reproducibility papers,which analyze prior works in different contexts, e.g., in
different domains, with different datasets, compared to different baselines.
* Resource papers that provide the foundation for future reproducibility, e.g., in terms
of new datasets or open-source software frameworks for the evaluation of recommender
systems.
* Methodology papers, which for example propose novel offline evaluation protocols or
provide validated measurement scales for user-centric evaluations of recommendations.
* Reflective works, such as guidelines for reproducibility, surveys of reproducibility
levels, or theoretical reflections on our research methodology.
Other types of submissions related to the wider topic of reproducibility in recommender
systems are welcome as well. Please reach out to the track chairs at
reproducibility2023(a)recsys.acm.org in advance in case you are not sure if your work could
be a match for the reproducibility track.
Submission and reviewing
Similar to the main track of ACM RecSys ‘23 the review process is double-blind.
Submissions therefore must not include the author names and affiliations, and the
code/data repositories should be set up to maintain anonymity as well.
We expect authors to provide all relevant materials that are needed to fully assess the
validity of the submission. In case of reproducibility studies involving computational
experiments, this would for example include all code, data, and installation instructions
that are needed to re-execute the experiments that are reported in the paper.
Each paper will be reviewed by members of the programme committee of the track. The review
criteria, among others, will include the following specific aspects:
* Novelty: What is new about the reported findings or insights?
* Impact: What is the expected impact of the reported findings or insights?
* Reliability: Is the work technically sound?
* Availability: Is the submitted work well documented and repeatable?
Each accepted paper will be included in the conference proceedings, and a selected number
of papers will be presented in a plenary session as part of the main conference program.
We encourage authors to create a companion website that hosts all relevant material.
Submission Guidelines
We accept long papers of up to 14 pages (excluding references) and short papers of up to 7
pages (excluding references) in ACM’s new single-column format, corresponding to the
categories of the call for papers for the main track. The length of a submission should be
commensurate with its contribution.
It is expected that at the time of submission, code and datasets used to reproduce the
experiments will be available under reasonably liberal terms and sufficiently
well-documented such that reviewers may consult that documentation as they conduct their
reviews.
SUBMISSION SYSTEM
Papers must be submitted via EasyChair at
https://easychair.org/my/conference?conf=acmrecsys2023reprodu. Note that papers intended
for the main track of ACM RecSys 2023 should be submitted to a different EasyChair
installation.
IMPORTANT DATES
* Abstract submission deadline: April 14th, 2023
* Paper submission deadline: April 21st, 2023
* Author notification: June 28th, 2023
* Camera-ready version deadline: July 26th, 2023
Deadlines refer to 23:59 (11:59pm) in the AoE (Anywhere on Earth) time zone.
PAPER FORMATTING
ACM’s archival publication format separates content from presentation in the Digital
Library to enhance accessibility and improve the flexibility and resiliency of our
publications. Following the ACM publication workflow, all authors should submit
manuscripts for review in a single-column format. Instructions for Word and LaTeX authors
are given below:
* Microsoft Word: Write your paper using the Submission Template (Review Submission
Format). Follow the embedded instructions to apply the paragraph styles to your various
text elements. The text is in single-column format at this stage and no additional
formatting is required at this point.
* LaTeX: Please use the latest version of the Primary Article Template – LaTeX to create
your submission. You must use the “manuscript” option with the \documentclass[manuscript,
anonymous]{acmart} command to generate the output in a single-column format which is
required for review. Please see the LaTeX documentation and ACM’s LaTeX best practices
guide for further instructions. To ensure 100% compatibility with The ACM Publishing
System (TAPS), please restrict the use of packages to the whitelist of approved LaTeX
packages.
Authors are strongly encouraged to provide “alt text” (alternative text) for floats
(images, tables, etc.) in their content so that readers with disabilities can be given
descriptive information for these floats that are important to the work. The descriptive
text will be displayed in place of a float if the float cannot be loaded. This benefits
the author as well as it broadens the reader base for the author’s work. Moreover, the alt
text provides in-depth float descriptions to search engine crawlers, which helps to
properly index these floats. Additionally, authors should follow the ACM Accessibility
Recommendations for Publishing in Color and SIG ACCESS guidelines on describing figures.
Should you have any questions or issues going through the instructions above, please
contact support at acmtexsupport(a)aptaracorp.com for both LaTeX and Microsoft Word
inquiries.
Accepted papers will be later submitted to ACM’s new production platform where authors
will be able to review PDF and HTML output formats before publication.
ANONYMITY
The peer review process is mutually anonymous (double-blind). This means that all
submissions must not include information identifying the authors or their organization.
Specifically, do not include the authors’ names and affiliations, refer to your previous
work in the third person (e.g., “Zhang and Di Noia (2023) recommended that RecSys
submissions be anonymized by referring to the authors’ prior work in the third person.”),
and avoid providing any other information that would allow reviewers to identify the
authors, such as acknowledgments of individuals and funding sources. However, it is
acceptable to explicitly refer in the paper to the companies or organizations that
provided datasets, hosted experiments or deployed solutions if there is no implication
that the authors are currently affiliated with the mentioned organization. Reviewers are
instructed not to search for tech reports, pre-prints, and other information about your
research. Your responsibility is focused on making sure that the paper submission itself
does not reveal your identity as author.
ETHICAL REVIEW FOR HUMAN-SUBJECTS RESEARCH
ACM RecSys expects all authors to comply with ethical and regulatory guidelines associated
with human subjects research, including research involving human participants and research
using personally identifiable data. Papers reporting on such human subjects research must
include a statement identifying any regulatory review the research is subject to (and
identifying the form of approval provided), or explaining the lack of required review.
Reviewers will be asked to consider whether the research was conducted in compliance with
applicable ethical and regulatory guidelines.
We encourage authors to consider further ethical implications and broader impacts of their
work, and to discuss these in an appropriate section of their papers; “A Guide to Writing
the NeurIPS Impact Statement” provides non-binding guidance on some of the kinds of things
authors may wish to consider.
ORIGINALITY
Each paper should not be previously published or accepted to any peer-reviewed journal or
conference, nor currently under review elsewhere (including as another paper submission
for RecSys 2023). Papers published in workshop proceedings may only be submitted if the
RecSys submission includes at least 30% new content; such papers must also reference the
original workshop paper on the submission form (but not in the anonymized paper).
PLAGIARISM
Plagiarized papers will not be accepted for RecSys 2023. Our committees will be checking
the plagiarism level of all submitted papers to ensure content originality using an
automated tool.
If you reuse non-novel text from a prior publication (e.g., the description of an
algorithm or dataset), please be sure to cite the prior publication as the source of that
text. If you have questions about reuse of text or simultaneous submission, please contact
the program chairs at least one week prior to the submission deadline. Please refer to the
ACM Publishing License Agreement and Authorship Policy for further details.
Papers violating any of the above guidelines are subject to rejection without review and
cases may be referred to the ACM Publications Ethics and Plagiarism committee for further
action where warranted.
PATENTING
Please take note that the official publication date is the date the proceedings are made
available in the ACM Digital Library. This date may be up to two weeks prior to the first
day of the conference. The official publication date affects the deadline for any patent
filings related to published work.
SIGCHI Submitter Agreement
RecSys 2023 is a SIGCHI conference and making a submission to a SIGCHI conference is a
serious matter. Submissions require time and effort by SIGCHI volunteers to organize and
manage the reviewing process, and, if the submission is accepted, the publication and
presentation process. Thus, anyone who submits to RecSys 2023 implicitly confirms the
following statements:
1. I confirm that this submission is the work of myself and my co-authors.
2. I confirm that I or my co-authors hold copyright to the content, and have obtained
appropriate permissions for any portions of the content that are copyrighted by others.
3. I confirm that any research reported in this submission involving human subjects has
gone through the appropriate approval process at my institution.
4. I confirm that if this paper is accepted, I or one of my co-authors will present the
paper at the conference, either in person or through a conference-designated remote
presentation option. Papers that are not presented at the conference by an author may be
removed from the proceedings at the discretion of the program chairs.
Reproducibility Chairs
* Hui Fang, Shanghai University of Finance and Economics, China
* Dietmar Jannach, University of Klagenfurt, Austria
* Magdalini Eirinaki, San Jose State University, USA
Email: reproducibility2023(a)recsys.acm.org