This is the line that is called from the ItemInfoPanel at least that you need to look at:
<grid name="parts" rows="1" cols="6" pos="156,-272" cell_width="75" cell_height="75" repeat_content="true" controller="PartList" visible="{isnotcomparing}">
<item_stack name="0"/>
</grid>
This calls the item_stack section of code in the controls.xml. You need to duplicate this code in the controls.xml, creating a new control, something like item_stack9 for instance.
Modify this new control item_stack9 so that everything is resized as you like.
Then modify the calling code in the ItemInfoPanel so that it calls item_stack9 instead of item_stack.
<grid name="parts" rows="1" cols="9" pos="156,-272" cell_width="50" cell_height="50" repeat_content="true" controller="PartList" visible="{isnotcomparing}">
<item_stack9 name="0"/>
</grid>
The reason other panels get messed up is that multiple windows call the item_stack control, so any changes to that control will affect all the other windows.