Conferida na fonte há pouco
Com a sua conta Joonow (grátis), você segue para o site da empresa e a vaga fica nas suas candidaturas.
Nenhuma empresa séria cobra taxa, curso, exame ou uniforme. Desconfiou?
Descrição da vaga
About the company
Our client is an artificial intelligence company.
The role
Raydar is recruiting for this role on behalf of our client. Design and build test environments, evaluation harnesses and labeled datasets that measure how AI systems behave across multi-step tasks. The work combines reviewing system logs, spotting recurring errors and turning them into reusable test material. The role suits an engineer who wants to own problems from diagnosis through delivery.
What you'll do
- Construct simulated software environments that mimic the tools and workflows automated systems use.
- Build grading logic and evaluation systems that hold up when systems look for shortcuts.
- Create benchmarks whose scores track real-world results.
- Review interaction logs to find and measure previously unrecorded failure patterns.
- Turn findings into reusable datasets and test suites for internal improvement and external partners.
- Support data and training agreements alongside the founders while refining the evaluation tooling.
Requirements
What we're looking for
- One to seven years of relevant experience.
- Hands-on work with training data, post-training, benchmarks or evaluation for machine learning models.
- Judgment to tell which failures or data points are meaningful and valuable.
- A track record of building or contributing to benchmarks, datasets or environments used to train or test models.
- Skill in replicating common business software well enough that automated agents can interact with it reliably.
- Understanding of grader design, evaluation pitfalls and the ways systems game metrics.
- Prior work at a data vendor is relevant if it produced training material or model assessments, not expert labeling alone.
- A concrete example of building a benchmark, dataset, environment or evaluation system.
- Commercial awareness of what data and environments are worth.
- Ability to explain technical decisions and trade-offs in an interview.
Bonus points
- Found a real flaw in a widely used benchmark.
- Caught a model gaming a grader and fixed the grader.
- Built environments or benchmarks a leading AI lab used for training or evaluation.
- Startup founding experience or participation in a selective fellowship.
- High-level competitive achievement in mathematics, programming, debate or games.
- Understanding of how benchmarks and datasets support commercial model evaluation.
Benefits
Compensation and benefits
- Base salary: USD 150,000 to 300,000
- Equity
- Health, dental and vision coverage
- Retirement plan
- Flexible time off
Location and work model
- San Francisco, CA, United States
- On-site (in office)
- Full-time