I think I'm starting to understand what's going on.
You open the the inventory with 'I' which selects the EquipmentPanel. Then you open the Item Action panel which selects it and set EquipmentPanel as the previous panel. You then close the Inventory which automatically closes the Item Action Panel, And for some reason the EquipmentPanel opens up the Item Action Panel.
Is there anychance you have the "I" button bound to other inputs other than Toggle the Inventory? Perhaps it is also used to click for some odd reason?
Perhaps you'll have better control by opening the Inventory and Equipment panel directly in a custom script. This way you can choose to open/close them without opening the "previous" panel. That should give you a lot more control on how to open/close panels and when to open the gameplay panel.
I know of another community member that changed the Panel source code, to make smart(open/close) default to not selecting the previous panel. Perhaps that can help you too.
Note that the Panel scripts can always be overwritten with custom code or even completely removed. The InventoryGrid, Equipment, etc..UI does not require Panels to work, but if you choose that route you'll need to bind the ItemSlotContainers to the Inventory in code.
I hope that helps solve this problem