The basic idea is to make enchants much harder to obtain to make them more valuable, rare, and rewarding to obtain as opposed to the current "plop down an enchantment table you can get in half an hour, set up mob farm, grind exp, then wait on RNG" method of vanilla. Minecraft's typical exp cost for enchanted books can be used here, combining the total exp cost it would be for each individual book (so if a gem has Efficiency III and Fortune III on it, add the costs of what those books would be when adding this gem). Only one gem can ever be added to a gear (hence the quality attribute of gems) so players will want to be intelligent. Once infused gems are acquired, they can be applied to gear on an anvil. Would give more use to otherwise useless gold. You could add another layer to this by requiring gold to make gold casing to hold the gems while being infused. For example, trying to get Fortune-based infusion would require letting the gem get hit with a shulker projectile on the proper gem type with the proper cut, or having a charged creeper need to blow up near the proper gem with the proper cut to give the gem the protection enchantment infusion. Once a gem is polished or cut it would need to be infused in various ways to get desired enchantment properties. This is all easy enough to change in regards to what quality/cut/type matters for. The type of gem would determine what gear/items the gem can be applied to.
The size of a gem determines what types of cuts are possible in the gem cutting table and the cut chosen for a gem would determine what type of enchantments are possible (eg round cut = silk touch and emerald cut= fortune). As a simple example, a "quality 10" gem would be able to hold 10 gem infusions (such as Eff IV, Fort III, and Unbreak III = 10 total gem infusions). The quality of the gem would determine how many total "charges" of enchantment a gem could hold.
In addition to cut options, size could contribute to how many mistakes you could make before screwing up means a loss of gem. If you fail the gemcutting or exit the gemcutting table, the gem in question is lost. The gemcutting mechanic would be similar Thaumcraft research where you needed to "connect the proper dots" but is RNG each time you loaded a research paper. Gems could then be polished in a rock tumbler block for basic purposes (tier 1-2 enchants as well as making the vanilla diamond and emeralds for other purposes), or cut on a gem cutting table block for more advanced purposes (tier 3-5 enchants). I would add 5-6 gems to world gen of rarity rarer than diamonds, all gems (including diamond and emerald) would drop as "rough cut" gems with random properties of size, quality, and initial shape.
Instead of enchanted books as loot or villager trades, gem schematics could be found that help players learn gem cutting or give boosts to gem cutting to reduce failure. You can google "simple gem cuts" to get an idea of 5-6 basic gem shapes that could be used for gem cutting. An enchanting overhaul that disables vanilla enchanting table and enchanting books and instead uses gemcrafting/gemcutting and gem infusion to combine into specific enchants that can then be added to gear.