Automated recording, classification, and evaluation of cooking actions using Viam robotics platform
This repository provides a complete pipeline for recording cooking videos and evaluating action classification performance using computer vision and AI.
recordScript/
├── 📹 recording/ # Video recording tools
│ ├── record_rgb_interactive.py # Interactive cooking coaching
│ ├── record_rgb.py # Basic video recording
│ ├── record.py # Recording launcher
│ ├── test_connection.py # Connection testing
│ ├── setup_environment.sh # Environment setup
│ └── output/ # Video and CSV output files
├── 🔍 evaluation/ # Automated evaluation pipeline
│ ├── automate_evaluation.py # Main evaluation automation
│ ├── setup_automation.py # Evaluation environment setup
│ ├── extract_and_align_classifier.py # Core analysis engine
│ ├── time_alignment_analyzer.py # Temporal analysis tools
│ └── evaluation_results/ # Evaluation outputs
├── 📊 data/ # Datasets, samples, and results
├── 📚 docs/ # Detailed documentation
├── README.md # This file
└── requirements.txt # Python dependencies
# 1. Setup evaluation environment
cd evaluation/
python3 setup_automation.py
# 2. Configure your video in Viam dashboard
# (Set the video path in replayCamera-1 component)
# 3. Run automated evaluation
python3 automate_evaluation.py
# 4. Review results in evaluation_results/ directory! ☕# 1. Setup recording environment
cd recording/
./setup_environment.sh
# 2. Test your Viam connection
python3 test_connection.py
# 3. Start interactive recording session
VIAM_CAMERA_NAME="overhead-rgb" python3 record_rgb_interactive.py- Manual video setup - configure video path in Viam dashboard
- Smart video detection - knows when videos finish playing
- Automatic log capture - captures and monitors server logs
- Comprehensive analysis - detailed performance metrics and insights
- Single session focus - process one video at a time for better control
- Guided cooking sessions - step-by-step action prompts
- Separate video files - individual recordings per action type
- Automatic logging - CSV timelines and action metadata
- Real-time coaching - cooking guidance and feedback
- Action classification - detects cooking actions with confidence scores
- Performance evaluation - compares against ground truth datasets
- Motion analysis - similarity scoring and temporal analysis
- Actionable insights - optimization recommendations
| Section | Purpose | Link |
|---|---|---|
| 🔍 Evaluation | Automated evaluation pipeline | 📖 evaluation/README.md |
| 📹 Recording | Video recording tools | 📖 recording/README.md |
| 📊 Data | Datasets and sample files | 📖 data/README.md |
graph LR
A[Record Videos] --> B[Automated Evaluation]
B --> C[Performance Analysis]
C --> D[Model Optimization]
D --> A
- Record cooking actions using interactive tools
- Evaluate classifier performance automatically
- Analyze results and identify improvement areas
- Optimize model parameters and retrain
- Repeat for continuous improvement
- Configure your video in Viam dashboard (replayCamera-1 component)
- Run automated evaluation pipeline
- Review detailed performance report
- Switch video in dashboard and repeat for next video
- Python 3.7+ with pip
- Viam SDK (
pip install viam-sdk) - OpenCV for video processing
- viam-server installed and in PATH
- Valid Viam robot with camera component
- Cloud-managed machine configuration
replayCamera-1component for video replay- Action classifier modules installed
# 1. Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# 2. Upgrade pip and install dependencies
pip install --upgrade pip
pip install -r requirements.txt
# 3. Validate setup (evaluation)
cd evaluation && python3 setup_automation.py
# Or validate setup (recording)
cd recording && ./setup_environment.shAlternative: Use the automated setup script
# Run the setup script (includes all steps above)
./setup_environment.sh- ✅ Session analysis with detailed metrics
- ✅ Action detection timeline with confidence scores
- ✅ Performance insights for optimization
- ✅ Timeline exports for debugging and review
- ✅ JSON reports for integration with other tools
- ✅ High-quality video files for each action type
- ✅ Detailed action logs with timestamps
- ✅ Metadata exports for training and evaluation
- Evaluate action classification models
- Generate training datasets
- Analyze model performance across different scenarios
- Optimize confidence thresholds and parameters
- Continuous evaluation of deployed models
- Performance tracking over time
- Automated quality assurance
- Integration with CI/CD pipelines
- Learn computer vision and robotics
- Understand action recognition pipelines
- Hands-on experience with Viam platform
- Real-world AI/ML applications
Common Issues:
| Problem | Solution |
|---|---|
viam-server not found |
Install viam-server and add to PATH |
Machine ID not found |
Check Viam config file has valid credentials |
Config update failed |
Verify Viam API access and permissions |
Analysis failed |
Ensure scripts are in correct directories |
Need Help?
- Check the detailed documentation in
docs/ - Run setup scripts to validate your environment
- Review error logs in the output directories
Upgrading from manual evaluation?
Before (Manual Process):
# 😴 Tedious manual steps for each video:
# 1. Edit config file manually
# 2. Run viam-server
# 3. Wait and copy logs
# 4. Run analysis manually
# 5. Repeat...After (Streamlined):
# 🚀 Simple workflow:
# 1. Configure video in Viam dashboard once
# 2. Run automation script
cd evaluation/
python3 automate_evaluation.py
# 3. Review results - done!This is an active research project! Contributions welcome:
- Fork the repository
- Create a feature branch
- Test thoroughly with your setup
- Submit a pull request
[Add your license information here]
💡 Pro Tip: Start with the automated evaluation if you just want to analyze existing videos. Use the recording tools if you need to create new training data or test scenarios.
🎯 Ready to get started? Choose your path above and dive into the detailed documentation!