Third-party dependencies may introduce security risks to the software supply chain and hence yield harm to their dependent software. There are many known cases of malicious open source packages posing risks to developers and end users. However, while efforts are made to detect vulnerable open source packages, malicious packages are not yet considered explicitly. In order to tackle this problem we perform an exploratory case study on previously occurred attacks on the software supply chain with respect to observable artifacts created. Based on gained insights, we propose Buildwatch, a framework for dynamic analysis of software and its third-party dependencies. We noticed that malicious packages introduce a significant amount of new artifacts during installation when compared to benign versions of the same package. The paper presents a first analysis of observable artifacts of malicious packages as well as a possible mitigation strategy that might lead to more insight in long term.