Booz Allen
Human Performance Systems Engineering Internship
Summer 2024
During my internship, I had the opportunity to work with two teams.
Human Performance Team
There are five tactical athlete domains. I was focused on a research effort to expand the team's knowledge of the cognitive domain. The overarching research question was:
"How can we accurately model and predict cognitive fatigue?"
The Dataset
We found a smooth pursuit eye-tracking dataset from Harvard Dataverse containing features such as mental fatigue, physical fatigue, and alertness score. The goal was to predict mental fatigue (binary) of the participants using the features captured during the study such as total gaze deviation, marginal gaze deviation, and more. A visual of the dataset with each feature is shown below.
The testing setup, where participant would wear smooth-pursuit eye-tracking glasses and "type" words on the tablet screen using the movement of their eyes.
A graphic I made to visualize the order of data collection, how trials occurred, and which features were collected.
I began model development in the typical way: data exploration, normalizing and encoding features, and SMOTE oversampling of the mental fatigue feature. I then moved into a multiple model comparison. The goal here was to get the model that performed the best, with no constraints on computation time. Nonetheless, I tested models in order of generally increasing complexity.
A graphic of each model I tested. I made this schematic as a reminder of how each model works.
This was my first time using many of these models, so I made sure I fully understood and could explain each one before implementing them with scikit-learn. To see my full in-depth analysis, see the notebooks in my GitHub.
To compare each model I created a composite scoring algorithm that took into account accuracy, ROC AUC, weighted precision, wighted F1 score, and weighted recall. XGBoost was performing the best with a composite score of 93%. However, I wanted to see if there were any features I could engineer to boost my results further without overfitting.
I did a ton of thinking and a ton of whiteboarding. This was my favorite part of my entire internship because I could let my imagination run wild and think of anything that made sense and just try it! One real-world use case of the project was to figure out "the mental fatigue cliff" or point of no return for an individual. At what point is the person saying, "I'm not mentally fatigued, let's keep going..." versus their data saying, "Your gaze deviation is abnormal, time to take a break." Thinking about solving that question is how I devised the "Mental Fatigue Derivative" (MFD) feature.
.png)
Looking at the change in gaze deviation over the change in mental fatigue is how we'd be able to set a threshold of when to notify a user to take a break even if subjectively, they feel okay. Additionally, the new feature boosted model performance!
.png)
Final model performance after MFD feature engineering.
After packaging the final model for deployment, I left the team with steps to use this solution in the field with their own eye tracking glasses. Instead of just creating a hypothetical model, I wanted to take an end-to-end approach and show that this research effort can be used for real impact in the field.
Real World Use Case
Special thanks to Matt Krinn for his teamwork during this internship! At the end of the summer, Matt and I wrapped our project into an impactful presentation for Booz Allen leadership.
NASA Systems Engineering Team
While at Booz Allen, I wanted to take advantage of their learning opportunities. I completed multiple digital engineering and model based systems engineering (MBSE) trainings. I was then introduced to the Extravehicular Activity and Human Surface Mobility Program (EHP) Team. I created multiple document export templates for the team, so they could automatically export figures from the digital engineering tool suite into relevant documents for client review and approval. This saved the team approximately five hours per template and reduced costs for the company.
Special thanks to all my mentors and colleagues throughout this internship. I learned so much here, and it was an incredible experience!