Artificial Intelligence & Deep Tech
Developing Realistic Consulting Tasks for Training and Evaluating AI Agents
The client, an international provider of AI training data, supports a leading developer of large language models in improving the performance of its AI agents through reinforcement learning from human feedback (RLHF).
For this purpose, the client needs complex, realistic consulting tasks, each equivalent to several hours of consulting work, that reliably separate strong from weak AI agent outputs.
Every task must be rooted in genuine project experience, be based on realistic working documents, and be validated by human experts at every step. Automation tools support the work but never replace expert judgment.
Designing task scenarios based on real consulting situations, deliberately targeting known weaknesses of AI agents, including:
Building a realistic project environment of more than 25 input files per task, ranging from downloaded and disguised real-world documents to emails and interview transcripts, with deliberately embedded challenges
Writing a concise task prompt in the style of a real manager’s request and finalizing a reference solution (“golden answer”) that is fully traceable to the project environment
Developing and calibrating the grading prompt (“grader”) through test runs on models of varying capability
Conducting multi-stage quality assurance (self-QA, automated and human QA, cross-QA), as well as reviewing agent outputs and writing expert preference annotations
Developed consulting tasks that clearly spread AI agent results apart and expose weaknesses in judgment
Calibrated the grading logic so that the reference solution scores clearly above the best agent run and weak outputs are reliably identified
Provided in-depth expert feedback on agent outputs and grading behavior as a basis for aligning the client’s evaluation models with the judgment of experienced consultants
March 2021
The 2021 ranking ‘Beste Unternehmensberater [Best Management Consultants]’ by the German business magazine brand eins and the business data portal Statista has just been published.
T. K. HAMANN takes a top position in this with a shared second place in the field of ‘Sales, After Sales, CRM.’ In addition, we are honored for the sector ‘Auto & Suppliers’ and the functional area ‘Strategy Development.’
We are grateful to all our clients, the entire team, and business partners who have worked with us since our founding in late 2016. Together, we will continue to consequently pursue quality leadership in our disciplines.
For this year’s ranking, 2,334 colleagues determined the best management consultancies according to their perception by industry and functional practice area. Subsequently, assessments were obtained from the clients. For this purpose, 1,100 senior executives were provided with a list of the recommended consultancies for each industry and work area clients via online survey; another 234 senior executives (Head of Strategy, Head of Business Development, etc.) from DAX, M-DAX, S-DAX and Tec-DAX companies were specifically invited to participate in the survey.