No, de-activated plugins cannot slow down the WordPress site. Wordpress only loads the active plugins and ignores everything else.