While compound lifts help strengthen your core muscles, many lifting routines involve additional core accessory exercises. If you've got a home gym, your exercise selection may be limited. Here are some options that either require no additional equipment, or relatively inexpensive equipment. Rotating these exercises into your workout routine can help build a stronger core to help with stability in your compound lifts.
You may not have an ab wheel already, but they can be found for under $25 and make for a great investment for your home gym. Ab wheel rollouts are quite difficult even for experienced athletes, but can be made easier by starting with your knees on the ground instead of your toes. Variations can also include rolling slightly to the left or right in order to activate your obliques in addition to your abs.
I really like reverse crunches as an alternative to normal crunches, as they keep your head and neck in a more neutral position. If you already have a flat bench for bench press, you can use that bench and hold on to the bench above your head in order to get in the right position. If you don't have a bench, you can also do these from the floor. These work well as superset exercises with compound movements.
While most core accessory exercises focus on abs, it can be useful to include exercises that focus on obliques. Side bends are great for your overall core development. If you have dumbbells, you can hold a single dumbbell to your side and do a set, followed by switching hands and doing the other side. If you don't have dumbbells but have weight plates with handles, they can work well for these.
Performing a hanging leg raise has a double effect of working your core as well as stretching your shoulders and arms. You can do these with your legs extended, or you can tuck your knees to make this slightly easier. Some pull-up towers have arms where you can hold yourself up instead of hanging, which can be a little more comfortable. These hanging leg raises can also be done anywhere you have a bar or handle that lets you hang freely, so you can perform them throughout the day.
Lots of core exercises involve abdominal contraction, but it's good to perform planks as an isometric static exercise. This kind of static holding is more closely related to compound movements, where you want your core to hold steady and brace while other muscles are moving. The difficulty of planks really ramps up the longer you hold it. Even just doing one more second than last time is a great goal, since before you know it you'll be holding for multiple minutes.