I don't dispute the fact that disenfranchising the Jewish citizens of Germany before the war, and looting and destroying their property, proved wasteful. How much is the question, however, and if the figures you provided are correct, the answer appears to be "not much". To give but one example, the military spending of Germany is estimated at 53 billion RM in 1940, 71 in 1941, 91 in 1942 and 112 in 1943. A loss of 1 billion RM per annum barely registers. Even your top estimate of 5 billion RM for occupied Europe is still but a fraction of the total, which in any case couldn't have been entirely redirected to German war production in the first place. Lastly, a large number of Jewish prisoners were used in economic activities, so they did not entirely "go to waste".Now, you remove the jews from the economy, you remove that production of wealth. 1 billion RM per annum for Germany alone. Then Germans go and do that for the entirety of Europe. In ocupied teritorries by brute force, in vassal lands by "persuasion". Most dictator-underlings go along to get along, some (Horthy) murmur a faint protestation, but go along nevertheless. Only Mannerheim and Franco laugh at such imbecility.
Regarding the Holocaust in itself, it probably was somewhat more wasteful, but also had some "benefits", as the liquidation of the Jewish citizens of Poland during 1942 illustrates: it freed up foodstuffs for the German population. The same would have applied, on a much larger scale, to the "Hunger Plan", which meant to starve out Russia's urban population to redirect agricultural production to the German Grossraum. For the Nazis, mass murder played both an ideological and a practical role, and often both at the same time.
Obviously, I am not condoning their atrocities, and even if the Holocaust or the planned genocide in Russia held some practical utility, I am not arguing that it was just a sensible, if cold-hearted, set of policies rendered necessary by the war. In the context of the Nazi regime's plans for a forcible reordering of the world, however, I don't think that their usage of extreme violence was in itself self-defeating - if anything, it was necessary for the achievement of their far-reaching goals, the grandiosity of which proved, IMO, a lot more decisive in their defeat than their wanton acts of brutality.