I recently graduated from computer science and am in my first year of doing practical work as a research fellow. I deal with water quality data for this project, but I don’t feel like I’m fully utilizing the data—rather, it feels like I’m merely graphing variables and performing simple statistical analysis. I must admit that I work alone most of the time. Most of the knowledge I gained while in college is not applicable to my current work, especially when it comes to data analysis. All of the studies I conduct are based on ongoing online research, but I frequently feel that, when it comes to time series analysis, I’m not doing enough other than trend estimation, correlation analysis and some statistical tests.
Yes, that is a typical emotion.
I firmly believe that more imaginative feature engineering can always be done. However, there are also declining returns and temporal limits that are nearly constant.
Could you elaborate on the reasons behind your perception that your analysis is lacking? Is it a lack of experience using time series techniques? Do you think there are features lacking or that the data set is too simple?
Indeed, job dissatisfaction is widespread in practically all fields. Since the introduction of foundational models like GPT, Dalle, etc., I have been a data scientist. The majority of the work consists of developing these models’ APIs for diverse use cases. However, I never stop honing my craft through open-source projects, fine-tuning foundational models, etc.
You can work on challenging projects, freelance, or even take a job other than trend estimates, correlation analysis, and certain statistical tests if you wish to step outside of your comfort zone.
In other words, if the ML portion yields positive results, your analysis was done well enough, or at least well enough.
My model was not doing well, so I had to go back to the chalkboard recently. What was the point of doing a more in-depth investigation if it was operating rather well? My time will be wasted on it.
I’m not saying be lazy; rather, I’m saying complete your EDA, figure out what you need, and often one round is sufficient. Iterations are sometimes necessary.
What matters is the value that the analysis has on the subsequent actions. If there are “downstream” users who would benefit from different approaches by all means branch out! If your user base is happy with your current output, figure out how to do it better. But remember that we don’t do analytics for the sake of analytics. Business value always has to be there.
Since most work is not at the level of a rocket scientist, it is perfectly acceptable to forgo a complex model or analysis. Additionally, as another commenter pointed out, the business team and management are more concerned with how much money the project saves or produces than with the model, techniques, or measurement metrics you choose.