Don't you get it steve?

When the jewel is on the equipment, you can upgrade it w/o removing the jewels.

When you remove a level 4 or higher jewel, you get charged gems to do so.

A curse? no not at all. it makes you decide whether to use the gems now, or wait and accumulate 5 of them to merge into a higher one and use them later.