Overview
The GenAI Project Evaluator assesses team submissions for Generative AI projects against the project Software Requirements Specification (SRS) and a 100-mark evaluation rubric. The evaluator verifies whether the submitted solutions work as expected, rather than simply checking whether features are present in the code. This includes running the application, testing prompts and AI-generated outputs, validating results against independent Python-based rules, checking source references, and testing the system with unseen inputs and edge cases.
The evaluator must provide fair, consistent, and evidence-based marks and feedback for each project. The total number of projects to be evaluated is 12.
Key Responsibilities
The evaluator follows a structured process from receiving the project submission to recording the final marks and feedback. Each project must be assessed against the SRS and the evaluation criteria listed below.
Key responsibilities include:
Evaluation Areas
The evaluator scores ten criteria, with a total of 100 marks.
Criterion
What the evaluator checks
Functionality & SRS Compliance
All functional requirements are implemented and working. Required workflows produce the expected results and handle valid and invalid inputs correctly.
GenAI Integration, Prompting & Structured Output
Correct integration with the selected GenAI model or API; effective prompts; appropriate handling of model responses; consistent output structure; and suitable handling of incomplete, incorrect, or unexpected responses.
Python Ground-Truth Validation & Rule Engine
Independent Python-based validation of AI-generated outputs; correct implementation of business rules; identification of incorrect AI decisions; and clear separation between AI generation and rule-based verification.
Source Grounding, Traceability & Hallucination Control
Responses are supported by approved source documents or reference data; relevant evidence and source references are provided; unsupported claims are identified; and the system avoids inventing facts or conclusions.
Dual-Pipeline Comparison & Hidden Evaluation Testing
AI-generated results are compared with independent Python validation results; discrepancies are identified and reported; unseen test cases and edge cases are handled correctly; and the system is ready to process new inputs.
Source Code Quality & Technical Implementation
Code structure, readability, maintainability, error handling, configuration management, reproducibility, and correct implementation of the overall workflow.
UI, Dashboard & User Experience
Ease of use, clarity of results, visibility of AI decisions and validation results, discrepancy reporting, filters, exports, and appropriate presentation of business insights.
Security, Prompt-Injection & Testing
Handling of direct and reworded prompt-injection attempts, unsafe or malicious inputs, sensitive information, exposed credentials, and other security risks. Evidence of meaningful testing is also reviewed.
Documentation & Published Blog
Accuracy and completeness of technical documentation, architecture and workflow diagrams, setup instructions, testing evidence, published blog, and demonstration video.
Plagiarism & AI-Usage Compliance
Completeness of AI usage disclosures, originality of submitted work, appropriate acknowledgement of AI assistance, compliance with project rules, and evidence of individual team contributions where required.
On-Time Submission
Final code, required documents, deliverables, and commits are submitted before the specified deadline.
Required Skills
The evaluator must have hands-on technical expertise in Generative AI applications, Python-based validation, and software testing, along with the ability to assess projects fairly and consistently.
